Serious Sam's Serious Thoughts

All About Python and Unicode

nospam@example.com (Misha) — Wed, 26 Dec 2007 01:10:18 +1100

I got caught on a weird issue comparing two strings in Python:

CODE:

/home/misha/python/wav2mp3.py:45: UnicodeWarning: Unicode equal comparison failed
to convert both arguments to Unicode - interpreting them as being unequal
if candidate == outname.lower():

I took a while before I could figure out what the problem was, but only after having a more or less thorough read through this site.

Two weeks before I started writing this document, my knowledge of using Python and Unicode was about like this: All there is to using Unicode in Python is just passing your strings to unicode()...

The "weird issue" was that the LHS in my comparison was a UTF string, while the RHS was a UTF-8 encoded string. My lame print debugging statements did nothing to highlight the issue, as print was correctly interpreting both the encoded and un-encoded strings. The repr() method proved quite useful in this case.

MinGW hanging on compilation

nospam@example.com (Misha) — Mon, 17 Dec 2007 11:08:40 +1100

Recently, I've had to install MinGW and play around with it. When building a project, the compiler seemed to hang after a couple of seconds, and I had to kill the process. I could restart the build, but after a couple of seconds it would hang again. Googling the problem yielded a surprising fix:

From http://lists-archives.org/mingw-users/03939-as-exe-hanging-intermittently-solved.html:

QUOTE:

When I compile programs - even trivial test programs - it usually works
fine, but some of the time gcc/g++ hang indefinitely. A bit of
investigation with Process Explorer revealed that the instance of as.exe

exec'd by gcc is always running when the hang occurs. `ps -aW' also shows it, but interestingly it has a pid of 0, with only the winpid being non 0. Process Explorer suggests that `as.exe' is not using any CPU time, nor is its memory allocation changing at all. I strongly suspect it's deadlocked somewhere. With more testing I found that the hang occurs whether as is run from gcc, or directly from the shell.

QUOTE:

Killing LVPrcSrv.exe does not cause hung as.exe processes to resume execution, but after it is killed new as.exe processes will not hang.

Cross-compiling pwsafe for ARM

nospam@example.com (Misha) — Sat, 27 Oct 2007 22:12:33 +1000

I've spent a good part of this weekend playing around with my shiny new Sharp Zaurus SL-C3200. I'm not going to give it any more of an introduction, because it would be impossible to top what this guy has written.

Instead, what I'm going to focus on is cross-compiling one of the apps I consider quite useful -- pwsafe. Pwsafe is a password manager (originally for Windows) which has clones for Linux, which is great for me, since I use Windows at work, and Linux at home. It stores passwords in an encrypted database which you can take with you wherever you go.

So, to put this thing onto a Zaurus, you need a couple of things to start with:

Embedix SDK (Embedix-sdk.tgz) for cross-compiler, libraries, etc.

OpenSSL source (openssl-0.9.7d.tar.gz) -- libssl and libcrypto are pre-requisites for pwsafe. Get 0.9.7 because that still has configure targets for ARM (newer versions seem not to have this).

Pwsafe source (pwsafe-2.0.0)

Standard Linux build tools (autoconf, make, etc).

Build and install the Embedix SDK. By default, it will go into /opt/QtPalmtop.

Now, you will need to build and install OpenSSL. I used these instructions to guide me. I couldn't really follow them to the letter, though -- something seems fishy with their patch file.

Anyway, configure the library for ARM, specifying the preferred install directory. Use the ARM cross-compiler:

CODE:

export PATH=/opt/Embedix/tools/arm-linux/bin:$PATH
./Configure linux-elf-arm --prefix=/home/misha/zaurus/arm

You'll need to hand edit the Makefile (or, if you're feeling lucky, try and patch it). All I had to change was add -ldl to the list of libraries to use.

CODE:

EX_LIBS= -ldl

All should be good to make and install the OpenSLL library now.

Now, it's time for pwsafe.

Open acinclude.m4 and surround jm_CHECK_TYPE_STRUCT_UTIMBUF with the M4 quotation marks (`[' and `]'). My aclocal bitched about this and led to a malformed makefile being generated at the end.

Open configure.ac and replace AM_SYS_POSIX_TERMIOS (apparently, this is obsolete) with AC_SYS_POSIX_TERMIOS. Without this, the configure script had problems with detecting some glibc stuff for me.

Configure

CODE:

CXXFLAGS=-fpermissive ./configure --host=arm \
--with-openssl-dir=/home/misha/src/zaurus/openssl-0.9.7d/dist --with-readline=no --with-x=no

You need -fpermissive because otherwise the compiler bitches

You need --with-readline=no if you're like me and can't be bothered building readline

You need --with-x=no if you're building this for QTopia, which doesn't have an X server

At this point, all you have left to do is make and deploy to the Zaurus. Libs (libssl and libcrypto) go in /opt/QtPalmtop/lib and the binary (pwsafe) goes into /opt/QtPalmtop/bin.

I'll wrap this up in an ipk a bit later to make deployment easier.

New Japanese dictionary for ZBEDICT

nospam@example.com (Misha) — Thu, 18 Oct 2007 00:06:05 +1000

Check it out here. It's always so hard to resist writing code...

PAdict and Palm T|X

nospam@example.com (Misha) — Sat, 29 Sep 2007 16:46:35 +1000

I've recently forked out some cash and bought myself a new Palm T|X to replace my aging second-hand Tungsten E. It finally arrived on Friday. Everything was good until on Saturday morning, I tried starting up PAdict and got the error message -- "The last PAdict session was not terminated properly." I was a little bit surprised, but hey, this happened a few times on my old Tungsten E, so I wasn't too alarmed and pressed the OK button to proceed. I went through a moment of silent shock when instead of starting PAdict as normal, the Palm just rebooted. Trying to start PAdict again yielded the same results.

Googling didn't come up with all too much, with the exception of one user complaining about the same problem.

QUOTE:

Even PAdict sometimes goes down (that's especially ugly, since its crashes require full reinstallation of the app, although the authors can't understand why).

This pissed me off -- PAdict is the main reason why I use a Palm in the first place. It's a great Japanese dictionary, and, what's better, it allows Kanji-illiterate people like me to get by with it's Kanji recognition feature. It's awesome.

Things were looking rather grim. I wasn't particuarly in the mood for reinstalling PAdict everytime it has a crash. I didn't particularly feel like going back to using the Tungsten E after splurging all this money on a T|X. So off I went, looking through the PAdict bug database to see if this has been addressed or at least reported.

This bug report saved my Saturday. To start off with, it's the sort of bug report any developer dreams of. And while it was reported on a Treo, it was a similar issue.

QUOTE:

... you should be able to fix the problem by deleting the "KJPADScratchPadDB" database using a tool like FileZ.

I happened to have my Palm plugged in to my Linux desktop while I was reading that, and decided to give the work-around a go using pilot-link:

QUOTE:

pilot-xfer -p /dev/pilot --delete KJPADScratchPadDB

It worked like a charm... PAdict works again. I ended up installing FileZ as well, for cases when PAdict tanks while I'm on the road.

Kuril Islands Dispute on MSN Encarta

nospam@example.com (Misha) — Thu, 27 Sep 2007 13:19:47 +1000

Recently, I've resumed spending some of my time researching the Kuril Islands Dispute. I've come across the following article from MSN Encarta:

QUOTE:

Kuril Islands (Japanese Chishima-rettō, Russian Kuril'skiye Ostrova, from the word kurit, "to smoke")

Smokin... great to know that they are getting their information from a quality source. I'm surprised no-one over there had the initiative to look up the Kuriles on Wikipedia to see the real origins of the name.

Polonez (Полонез -- Чиж и Ко)

nospam@example.com (Misha) — Wed, 26 Sep 2007 20:01:03 +1000

My first attempt to record this was foiled by the premature arrival of the pizza dude. There was no stopping me this time (even mixing up the lyrics didn't stop me). Check it out here.

Programming can ruin your life

nospam@example.com (Misha) — Thu, 13 Sep 2007 22:05:29 +1000

A colleague at work sent this this link around. I must admit, I felt a slight shock while reading it. "Wow, this is kind what is happening to me," I thought. At the same time, I entertained brief thoughts of relief... "Whew, at least I'm not a complete geek". At times like this I generally seek salvation in sports, or hobbies that have as little to do with programming as possible.

I remember second year uni, where in a desperate attempt to not burn out in computing I took Japanese as an elective. It helped, and I made it through second year and managed to eventually graduate, but I can't help thinking that learning a foreign language isn't completely unrelated to programming... Now that I think about it, the majority of the things I do are in someway programming related. And even if they are not, I find myself taking a programmer's approach to them. While it works, and I tend to achieve most of the goals that I set, it's kind of concerning to acknowledge that my choice of profession has resulted in a mind-altering experience.

Then again, if I think back to my days as a gamer back in high school, I distinctly recall a brief period of time where I was obsessed with completing a stage of a game in the most perfect way I could find. If I screwed up, then I re-loaded from a "perfect" save-point and tried again. I do remember acknowledging to myself that in a way it was ridiculous, it didn't really matter that much, cause I was having fun. I find some consolation in that those those days were years before I even thought about programming -- so, in a way, it could be that my behavioral pattern matched my career choice, as opposed to my career choice altering my behavioural patterns...

If that is so, then the title of the above-mentioned article doesn't really apply to me, because in essence my life was "ruined" well before programming managed to get it's claws on it .

As I'm writing about this, I'm chatting with my father, who has also been blessed with the experience of treating programming as a profession. "That's like saying that life leads you to death," was his opinion of the article. While I agree, it was an interesting and thought provoking read.

Notes of a Japanese POW

nospam@example.com (Misha) — Thu, 13 Sep 2007 21:36:41 +1000

In my leisurely nightly wanderings across the web, I've come across a surprisingly moving site. It is told, mostly in pictures, from the eyes of a Japanese POW and details his captivity in the USSR after WW2. Although it is related to war, it sends a message of peace. I'd recommend anyone have a look at it.

Recordings on YouTube

nospam@example.com (Misha) — Thu, 06 Sep 2007 18:00:46 +1000

Well, this was bound to happen eventually. One of my mates, Roman and I recorded some music and put it up on YouTube for the whole world to see. You can check it out here.

Haven't received any hate mail yet...

Bug in System.Uri constructor?

nospam@example.com (Misha) — Tue, 04 Sep 2007 14:09:28 +1000

I'm using the System.Uri class to clean up some the URLs we're passing around, e.g. compacting http://hostname//somefolder//somefile to http://hostname/somefolder/somefile.

The documentation clearly states that the Uri constructor takes care of this: MSDN.

This works fine on my machine. However, when I try this on our server box (Win 2003), I just get back the string I passed to the constructor.

I've checked the .NET versions and the assembly versions on both the target machine and my workstation. They match.

Here's the code:

CODE:

using System;

namespace sandpit
{
    class Program
    {
        public static void Main(string[] args)
        {
            Type type = typeof(Uri);
            Console.WriteLine(type.ToString());
            Console.WriteLine(type.Assembly);
            Console.WriteLine(type.AssemblyQualifiedName);
            Console.WriteLine(type.UnderlyingSystemType);
            Uri uri = new Uri(args.Length == 0 ? "http://hostname///
somefile.txt" : args[0]);
            Console.WriteLine(uri.ToString());
        }
    }

}

I've posted this on Google Groups but so far, to no avail.

The operation is not valid due to the current state of the object

nospam@example.com (Misha) — Thu, 16 Aug 2007 17:39:16 +1000

I've recently become re-united with an old friend -- that's right, I'm back to working with SharePoint. Since this time I'm armed with a little bit more time and resources, I'm hoping it will be less painful than it was last time.

The first major snag I've hit so far is the issue of how to deploy custom web services on the SharePoint server. The docs from the SDK recommend the approach of adding the web service to SharePoint so that it can be accessed from _vti_bin/servicename.asmx. From my relatively inexperienced point of view, this is good for a number of reasons, the main one being that the custom web service runs as part of SharePoint, and things like permissions, privileges and access control seem to be handled for you. Great.

There's a catch though. The main one is that you need to dump the WS assembly into the server's GAC. There are other pains in getting the disco and wsdl files set up for your custom WS, but the SDK walks you through those.

An alternative to the recommended approach is to deploy your custom web service to an IIS web site separate from SharePoint (obviously, this is still on the SharePoint server). This seems to opens up a whole new can of worms with privileges. The fun begins.

The problem I ran into was an "The operation is not valid due to the current state of the object" exception being thrown when adding files or updating items. Google saved the day rather quickly -- http://howtocode.blogspot.com/2007_06_01_archive.html.

QUOTE:

You must not call the SPListItem.Update inside the RunWithElevatedPrivileges block. Instead you should only instantiate the SPSite or SPWeb there and call Update afterwards...

The strange thing is that this seemed to work last year, when I was bashing my head against the WSS Beta TR2. I guess things change...

Motobook

nospam@example.com (Misha) — Wed, 18 Jul 2007 22:23:48 +1000

I've finally uploaded a small app I've been writing in my spare time -- get it here.

QUOTE:

This is a command-line utility that can be used to backup and restore the phonebook on a Motorola phone. I was motivated to write it as I couldn't find a way to backup my phonebook without booting into Windows. I use it with my RAZR V3, and haven't tested it with any other model, so your mileage may vary.

Motorola stuff on Linux

nospam@example.com (Misha) — Mon, 14 May 2007 21:58:34 +1000

I've owned my trust Motorola RAZR V3 for a while now, and have gotten used to many of its' shortcomings (the sluggish GUI is my biggest gripe). One thing I've always been putting off is connecting it to my Linux desktop at home. Finally, yesterday I mustered the courage to go out and confront this momentuous task.

To be honest, I was rather dissapointed. Everything is so much easier than I expected... Uploading and downloading of files is handled by the moto4lin application. I use FC6, so I was able to get it just by using:

CODE:

sudo yum -y install moto4lin

If that doesn't do the trick for you, then Google is your friend.

I was able to download photos and other crap from my phone in minutes.

Excited at the prospect of writing my own apps for the RAZR, I also went to the Motorola dev site and signed up to use their SDK. Unfortunately, the SDK comes packaged as a Windows executable -- interestingly, they don't supply a Linux version of the SDK even for their Linux-based phones. Go figure.

Oh well, it was time for Wine to save the day once again. The SDK installer nagged about me needing Java SDK and JRE installed and ran away the first time I ran it. Tried installing Java 5 SDK -- the install looked a little shaky, but completed till the end -- unfortunately, the Moto SDK installer still bitched about me not having an installed JRE. I fired up regedit -- there were JRE entries in sight. In desperation, I tried installing Java SDK 1.4.2 -- the install bombed halfway through but left me with the required registry entries. I promptly used Wine to install the SDK and get at the goodies inside.

The SDK contained pretty much everything I expected -- more or less abundant documentation, plenty of samples. There are emulators there, too, but they don't seem to run very well under Wine. The executables fail to start with an error code of c0000135. I haven't looked into this error code, but there seem to be a couple of hits from Google on it. I'll follow it up later.

The first thing I really wanted to do was to chuck a sample app onto the RAZR. There are generally two ways of doing that:

USB transfer

WAP transfer

I don't have a GPRS modem, so that kind of narrowed down the options for me eventually.

First I tried using Midway, from Motorola's SDK through Wine. It bitched about COM ports being unavailable. Fine.

Using `dmesg | tail', you can find which dev corresponds to the modem on the phone. It my case, it was /dev/ttyACM0.

Allow Wine to access the device through a COM port of your choosing. I chose 3.

CODE:

ln -s /dev/ttyACM0 ~/.wine/dosdevices/com 3

I then fired up Midway again -- it still wouldn't talk to the device, so I tried running it as root. Voila! I could connect.

Unfortunately, progress came to a brief halt there. I was able to open up the JAD files from the SDK samples, but couldn't upload them to the phone. It would always time out. Some googling revealed that you need the follow the following steps in strict order:

Start with the USB cable disconnected

On your phone, go to Settings/Java Settings/Java App Loader

When you see `Insert cable now', do what it says.

Open up the COM port for Wine (see above)

Fire up Midway through Wine

If you're not into Wine, there's a really cool command line app that you can use instead of Midway. It's located here. It also comes with a GUI version, but that requires QT4 and I couldn't be bothered installing it.

Well that's about it for now. When I have nothing better to do I might actually write something trivial for my phone. A metronome to help me with my back exercises would really be super.

Kanji study

nospam@example.com (Misha) — Fri, 06 Apr 2007 01:06:36 +1000

I managed to goof up and leave my Palm at work for the Easter break... Damn. I've really grown quite used to it in helping me remember the Kanji from Heisig's book. Since I had absolutely grandiose plans for Japanese study over the weekend (and, as of 1am on Friday morning, I still do), I decided to look for an alternative study tool for the next few days... (I really don't feel like driving all the way to work just to fetch my Palm). So, I found this site -- I think it's really cool and deserves a mention. Basically, you register (for free) and as you progress through the Kanji list, you add flash cards. It keeps track of the Kanji that you remembered, and ones that are not. It's great for reviewing -- I foresee an exciting weekend...