The Stochastic Game

Ramblings of General Geekery

Visual Studio Express’ limitations lead to bad practices

Visual Studio Express only has a subset of what you can find in Visual Studio Professional, which makes complete sense, but two missing features actually prevent users from following best programming practices in my opinion. The whole point of Express is to let enthusiasts and students discover the joys of programming – so we might as well let them do it the proper way.

The first limitation is not being able to set an external executable as the “start action” when you debug. In Express, you can only run the debugger on an application project (console, UI, or web application). You can’t set a library project to be the startup project, nor can you attach the debugger to another process. This means that if you want to debug your unit tests, the unit test project must be an executable.

Thankfully, most unit testing frameworks have a console bootstrapper that you can use as your console application’s main loop, but it’s not ideal, and it probably doesn’t incite many people into writing unit tests because they have to figure all this out. More importantly, it breaks down when you can’t create console applications at all, like when you develop with Silverlight (although .NET 4’s assembly compatibility with Silverlight 4 may make things smoother here).

A way to get around that would be to use add-ins like TestDriven.NET, but Express also has a limitation that it doesn’t support add-ins (this actually got TestDriven.NET into some trouble at some point). Other ways to get around it would be to access Visual Studio’s command window, or drive it with VB macros, but Microsoft closed those “exploits” with Express 2008.

The only way to get around those limitations is to use the .NET Framework’s CLR debugger, which ships with the SDK and is a stripped down version of the Visual Studio shell featuring only the debugging features. The problem is obviously that it’s a bare bones debugger that’s completely disconnected from the IDE experience.

The CLR debugger is also the only way to get around the second of Express’ limitations… You’ve probably already encountered a program that does heavy processing on the UI thread instead of in the background, resulting in sluggish user interaction, progress report, and lack of a button for cancelling the operation. Well, it’s pretty difficult to implement it otherwise with Visual Studio Express, because it doesn’t support multi-thread debugging.

Sure you can put breakpoints in code that will run on different threads, and it will break correctly, but you won’t be able to access the “Threads” debug window because there’s none. This means you won’t be able to look at what’s running at the same time, so it’s pretty useless… and it’s a shame because the framework makes it so easy to use thread pools and background workers and such.

It seems critical to me that Microsoft add those missing features to Visual Studio Express as they expand the open-source .NET community with their own projects. It should be as easy as possible for anybody to participate – not just people with a $1200 MSDN subscription. But the reality is that most of those open-source projects aren’t usable on anything else than the Professional versions (which opens up another debate regarding the culture of the .NET community, but that’s for another time). Of course, people could still use other IDEs like SharpDevelop, but then what’s the point of having those Express editions? I’m sure Microsoft would be happy to keep all those students leaning on Windows and Visual Studio – as they should. So if Microsoft is committed to a free, functional and educational version of Visual Studio, I think they would have to fix the feature set appropriately.


My home media & entertainment setup

I was working on this article when I spotted that my friend Bertrand Le Roy posted on that very same subject so I’ll turn this into a reply to his. The new year seems like a good time for bragging about one’s home video setup, it seems.

First, you may notice that my setup is quite simple because I don’t have any audio gear. Yet. That’s because until recently, my apartments were too small for me to have any decent speakers.

Home Theatre PC (XBMC Dashboard)

HD PVR

I’m still using an HD PVR (bottom left in the photo above) provided by my cable & internet provider, unlike some people who built their own PVR or cancelled their cable subscription altogether. The reason is that up here in Canada, we don’t have a lot of options for streaming or downloading legal content yet. It’s particularly frustrating because we get channels like NBC and such that proud themselves in telling you, at the end of the show, how you can go on their website to watch episodes you missed… only to get a cold “sorry, this content is not available in your country”. Anyway…

So I have an HD PVR for recording shows. I never, ever, watch live TV. Having to wait through commercials is too painful. I’m looking forward for the time when there will be no TV channels anymore – just direct subscriptions and downloads from content producers, similar to how we don’t listen to music through the radio anymore. Well, some people still do, but I don’t, and that’s made possible by the new platforms. Now if only a Zune Marketplace could open in Canada… but I digress again.

Media Center PC

For the digital content we do have access to in Canada, and for my growing collection of converted DVDs and CDs, I have a Media Center PC.

Similarly to how I don’t watch live TV anymore, I’m in the process of not watching real DVDs anymore (all my CDs have already been converted a long time ago). Because yes, it takes too much time to locate the correct case, open it, remove whatever disc was previously in the tray, locate that other case, put the disc back, put the case back, put the new disc in, load it up, wait through the incredibly enraging and often non skippable segments including very long logo intros, anti-piracy shorts, and sometimes even commercials, and then wait for the stupid main menu animation to finish so you can click on “Play Movie”. I prefer to just locate what I want to watch in a list and press “OK”, thank you very much.

Wow, I keep digressing. Sorry.

Home Theatre PC (Close-Up)

Hardware

So this, above, is my Media Center PC (you can spot it at the bottom right in the first picture). I built it last spring, and if you want all the facts, here’s what’s inside the Antec Fusion V2 case:

  • Zotac motherboard with a GeForce 9300 chipset and (among other connectors) HDMI output.
  • Intel Core 2 Duo CPU at 3.06GHz.
  • 4Gb RAM.
  • Seagate Momentus 5400.6 160Gb (this is a laptop 2.5” hard drive, for minimum noise).
  • Lite-On iHES106-29 DVD/Blu-Ray.
  • Logitech Cordless Desktop S520 Wireless Keyboard and Mouse.

Home Theatre PC (Hardware)

For roughly the same price as a Mac Mini, I get a bit more power, more RAM, a Blu-Ray drive, 1080p HDMI output, wireless keyboard and mouse, an IR receiver that works with any remote, and the freedom to upgrade it over the next years (what I don’t get is WiFi or customer support though). Sure, the Antec case is huge and not so sexy, but it’s quiet as hell. The fridge from the kitchen is noisier than the PC, actually.

Software

The center piece is XBMC.

Home Theatre PC (XBMC)

Oh, God, this is one beautiful piece of software. And it becomes truly awesome when you slap a skin like Aeon on it (although I need to upgrade and checkout the new 9.11 default skin).

I’ve been using XBMC for several years, all the way back to my original modded XBox, and although I sometimes look back, I never find any other similar program that’s even half as good. XBMC is just the right balance between a user-friendly “it just works” and an open-source “tweak the hell out of it”.

Home Theatre PC (Zune)

For my music, I prefer to use the Zune Software as a player. This is another beautifully designed program – at least in the graphic department. It just makes any other player look like an ugly Excel spreadsheet.

To manage my music library, however, the best thing I’ve found so far is MediaMonkey. This program alone, which I bought in “gold” version, is the reason I don’t have a Linux installed on my Media Center PC. It makes it easy to manage and maintain a huge music collection when you’re obsessive about tags, album covers and file names.

Other programs include Boxee, which is useful for some things but not quite as appealing outside of the US.

File Server

The Media Center PC itself doesn’t store any data beside its OS and programs. All the data (music, pictures, videos) is stored on a Netgear ReadyNAS NV+ with four 1TB hard drives in RAID (which gives me around 2.7TB of space).

File Server

Note the USB hard drive next to it. It receives daily backups of the important data… but I’ll probably write another post sometimes about my data storage and backup strategy (it’s an even geekier bragging subject!).

The ReadyNAS has a decent community going on, and since it’s running some kind of Linux, it’s easy to mess around with it if you’re not afraid to lose your warranty.

Gaming

For gaming, I have the obligatory Xbox360 (featured on the left on the first picture), and a Wii I won at a raffle (about the only time I ever won something) (featured on the right on the first picture).

I don’t use my Xbox360 as a media extender or anything because it’s noisy and, unlike my Media Center PC, needs to be turned on and off (the failure rate on this console is bad enough that you probably don’t want to keep it running all the time!). I however recently bought a couple of videos off the marketplace on it so it may take a more prominent role in the future.

Bringing it all together

Because I obviously want to control all this from my couch without thinking about it too much, I have a Logitech Harmony 510 universal remote.

Logitech Harmony 510

This is not the kind of universal remote where you need to press some switch button every time you need to control a different device. It handles things per “activity”, which means all the buttons can be mapped to various devices so that, well, you don’t need to think about it – “Menu” displays the menu, “Volume Up” increases the volume, etc., whatever device that means controlling.

Voilà!

And that’s it! The next steps are obviously to add some nice sound system, and finish ripping all those DVDs (which includes fighting the region lock crap because I bought some of my DVDs back when I lived in Europe).


Exposing global variables in IronPython

Lately I’ve been playing around a bit with IronPython and how to embed it inside an application to add scripting features. I’m still figuring things out, but I had a hard time exposing global variables to the Python environment.

The idea was to expose a couple of .NET objects (mainly a few important managers/singletons from the app’s API) as global variables so that scripts could access and act on the important parts of the app (query the database, batch run actions, etc.).

At first, I exposed some objects as variables of my ScriptScope:

public void SetupScope(ScriptScope scope)
{
    scope.SetVariable("test_string", "This is a test string");
    scope.SetVariable("test_callback_method", new Action(TestCallbackMethod));
    scope.SetVariable("test_callback_function", new Func<string, string>(TestCallbackFunction));
}

The problem was that only interactive scripting would get access to those variables (I had a simple UI for typing commands to be executed on that ScriptScope). Using “test_string” in a function loaded from a module would result in a “name ‘test_string’ is not defined” error. Using either “import” or “global” would not fix it.

I then discovered the ScriptRuntime.Globals property, and tried to add something there instead.

public void SetupGlobals(ScriptEngine engine)
{
    engine.Runtime.Globals.SetVariable("test_global", "This is a test global variable.");
}

This didn’t quite work either, and was actually a step backwards: now I couldn’t even access this “test_global” variable from the interactive command line!

That’s until I tried the following:

import test_global

Now I could access my global variable! And using that import statement from my modules also successfully imported it into their scope. Yay!

I’m not quite sure why the import statement is working, and why the other things didn’t, but I’m a Python newbie so it’s not surprising. Reading the Python documentation, though, tells me “import” is used to import packages, not variables, so it remains a mystery to me… but at least, it works on my machine for now!


Some more contacts love

There’s been a lot of improvement in communications in the past few years, from better services to brand new ones, but I still feel like contact management is lagging behind. I mean, isn’t it important to be able to find how to contact somebody in the first place?

Here are a few things I think could be better.

People have a lot of email addresses

With all the storage space provided by modern email services, and philosophies like GMail’s that advocate not deleting your email, you may have pretty old messages in your account. For instance, mine goes back to 1999 (and sometimes I wish I had thought about keeping my email before then because it’s fun to read those old conversations). This means some of your friends and coworkers may have changed jobs and personal email providers a few times. Some of my contacts have up to a dozen email addresses, actually.

Yahoo! Mail and GMail offer a dynamic list of email addresses where you can add as many as you want for a person, which is good, but Live Mail only gives you a static list of “home”, “work”, “IM” and “other”, which sucks. GMail lets you describe each address as “home”, “work” or “other”, but suprisingly doesn’t let you define custom descriptions. Yahoo! Mail doesn’t have descriptions so you have to guess which goes where.

One thing missing in all 3 services, and that I strongly wished existed, is the ability to tag specific addresses as “deprecated”: if one of your friends isn’t using that old AOL address anymore, you don’t want it ever suggested to you when you compose a  new message. However, you still want this address to be in the system for when you search old conversations with that contact.

Identity Profiles

Going beyond all this, I sometimes wish email providers would evolve from the old contact model of “name/email/address/notes” probably defined in the early days of Lotus Notes or something. The same way OpenID or InfoCard have “identity profiles”, each with its own set of contact information (name, email, address, website, etc.), contact management could also feature such profiles.

“Work” and “Home” profiles would be the most obvious ones, and would let the user tie together a set of previously unrelated information: right now, contacts may have email addresses for work and home, and IM nicknames for work and home, but they’re in separate lists, with no way to tie them together. Besides, as far as I can tell, no email provider even offers the ability to tag an IM nickname as being for “work” or “home” anyway.

The identity profile paradigm could then be used in powerful ways by client applications. For instance, the “work” profile would be the first suggestion on weekdays between 9am and 5pm, but the “home” profile would take over on week-ends and week-day evenings.

Don’t notify, just change

You probably know how much of a pain it is to notify everyone you know that you have a new email address, home address, and/or phone number. In this age of feeds and automatic updates, it’s weird that there’s no fancy technology with a hard to pronounce name that would do just that. Still, several people like Douglas Purdy or Tim Bray have been thinking about something like the following for a year or two now (that’s a lot in internet age!), on which I expand a bit.

The idea would be to use something like the hCard standard, a combine it with some RSS or PubSubHubbub magic: people who want to contact you would pull or get pushed your information instead of storing a completely different (and possibly out-of-date) version in their address book. Since the information is defined by you (via your email provider, your own server, or some 3rd party like for OpenID), it’s always up to date. You only have to change it once in one place and everybody else uses the updated version from then on.

It would especially be awesome if that kind of technology was used by governments, banks, ISPs, phone and cable operators, and all those guys that you always have to write to whenever you move.

Also, if coupled with identity profiles, you could have scenarios where somebody going away on holidays would update his information by making his “on the move” profile the main one, and all the other ones (especially the “work” one) disabled temporarily. If the whole system is correctly designed, this would let us avoid those awful “Sorry, I’m in Hawaii until next month” emails, because the client application would already know all this before you send anything. It would also ideally work not only with email but other protocols as well (IM, etc.).

Clever guy needed

That’s pretty much it. Now I just need a clever guy to make this happen because, well, like most programmers, I’m lazy. Besides, somebody probably had all those ideas and more, already. Still, I’d like my GMail inbox to get better.


Fun with jQuery: the vertical “Coda” slider

Update: my personal website has, since this article, been redesigned and does not feature this technique anymore. The demo page is still available however. It was broken for a while, but should be working again now.

I recently published the new version of my personal website and you’ll have no problem figuring out that I had some fun with jQuery. Probably a bit too much fun, actually, but hey, a personal website is supposed to be just usable enough that you can contact the owner without hassle.

My first approach to the website was a mix of good practices and totally blasphemous process:

  • I wanted a simple website that only lists ways to get in touch with me (Facebook, LinkedIn, and all that social web stuff), and ways to follow what I’m doing online (my blogs, twitter updates, and all that other social web stuff). It would additionally have a contact form so that people could send me a quick message without any extra steps.
  • I also wanted a website with some funky jQuery shit going on. I wanted to learn the API along with vanilla Javascript.

After some research about what jQuery can and cannot do, and a few sketches on my trusted notebook, I came up with a totally revolutionary (in the post-Web 2.0/post-iPhone 21st century definition of the term1) idea: the vertical “Coda” slider!

This UI pattern has been popularized by Panic’s website for their Coda software. It features a panel in which pages slide in and out when you click on their title, displayed in some kind of list or menu bar. There’s a nice tutorial on “jQuery for designers that explains how to reimplement it. The only difference is that I wanted the pages to slide in and out vertically instead of horizontally. You may point out that the aformentioned tutorial features an option to do just that, but it’s not quite the way I wanted. I want it to look like a standard page until you click one of the navigation links, which is when it goes “shwoop” and you go “wow I did not expect that!”.

No need to go through a lengthy tutorial, as you can probably figure how it works by just looking at the code, but still, to make it easier on you I created a “demo” version of the page with placeholder content and none of the other bouncy crap going on. Go check it out if you want to steal some of it, although I still have a couple of little quirks to fix, especially with window resizing.


  1. Which means “not very revolutionary”. ↩︎


Some similarities between Apple and Steve Jackson Games

Apple is company whose boss is a guy named Steve who is, by reputation, quite charismatic but also a real asshole when it comes to working with him and using his intellectual property. Their main product gives them only a small fraction of the market, and its core of devoted fans can be loyal up to a rather fanatical point. This product is always set against the more popular product, which is seen as outdated, inferior, over-marketed, and riddled with product updates that break compatibility with silly new features. Flamewars about which product is better are frequent. Apple’s product supposedly covers everything you may need, although fans still usually spend large amounts of money to get add-ons and accessories. However, the other product is still the dominant one by far, and most beginners start with it. Ironically, Apple’s most successful product is a small and fun “side” product. It has seen several iterations and lots of additional products are available.

Steve Jackson Games is company whose boss is a guy named Steve who is, by reputation, quite charismatic but also a real asshole when it comes to working with him and using his intellectual property. Their main product gives them only a small fraction of the market, and its core of devoted fans can be loyal up to a rather fanatical point. This product is always set against the more popular product, which is seen as outdated, inferior, over-marketed, and riddled with product updates that break compatibility with silly new features. Flamewars about which product is better are frequent. Steve Jackson Games’ product supposedly covers everything you may need, although fans still usually spend large amounts of money to get add-ons and accessories. However, the other product is still the dominant one by far, and most beginners start with it. Ironically, Steve Jackson Games’ most successful product is a small and fun “side” product. It has seen several iterations and lots of additional products are available.


Experimental IronCow branches

I created 2 experimental branches for future versions of IronCow.

  • IronCow Mobile” is a branch that adds support for the .NET Compact Framework. Thanks to jwboer for the initial patch.
  • IronCow Local Search” is a branch that adds local search for tasks. We basically cache all the tasks in memory, and handle search queries locally, instead of sending a request to the RTM server and parsing the response markup. The lexical analysis and AST building of the search query is a bit dodgy, as I can’t get a proper tool like ANTLR to work with RTM’s search grammar (probably me doing something wrong), but it’s not too much of a problem right now since search queries tend to be quite short, and we already are significantly faster than a web request.

Check them out!


Target the .NET Compact Framework using Visual Studio Express

Microsoft only supports Visual Studio Professional for developing Windows Mobile applications and, more generally, code based on the .NET Compact Framework. You get nice things like application deployment and emulators and remote debugging and all. But if you just want to compile something against the .NET Compact Framework, for example to check that you’re using supported methods and classes, you can do that with Visual Studio Express.

Create a project in Visual Studio Express and open it in a text editor. In the first “<PropertyGroup>” node, add the following at the end:

   1: <NoStdLib>true</NoStdLib>

This will tell MSBuild to not include mscorlib.dll automatically, so we can make it use the Compact Framework’s version.

Next, re-open the project in Visual Studio Express, delete all the system references (System.dll, System.Xml.dll, System.Data.dll, etc.), and re-add them, only this time use the Compact Framework’s assemblies. You’ll have to directly browse to those DLLs, as they probably won’t show up in the default dialog.

Now rebuild you application. It should build against the Compact Framework. You can test that by adding an instruction that’s unsupported, like for example “Thread.Sleep(TimeSpan)”.


Migrating to Google Apps

Migrating from a regular public Google account (GMail, etc.) to a Google Apps account seems to be a hot topic among geeks. Lots of people did it and posted their experience on their blog, which is often helpful for the next ones to try it. Since I recently migrated my account too, I thought I’d share this here. The important difference is that most people only post how they migrate their email. I tried to post about a lot more than that, including how to migrate contacts and groups and filters and quick links and documents and all that. I also wrote a complete “pros & cons” section up front so you can check whether Google Apps is for you.

Pros and Cons

Pros

  • You can administer your domain using Google’s control panel. All the users in your group (probably your family, or some kind of circle of friends) have a common ground to communicate on (chat integrated in mail, calendars, documents sharing, etc.). Scott Hanselman has a few pros & cons on that subject in his article about migrating his whole family to Google Apps, and the follow-up article . Note that these articles are outdated in a few places (some things have been fixed, some methods of backup/transfer don’t work anymore, some pros/cons are not valid anymore).
  • Your integrated GMail chat will now make you appear online as “username@domain.com” instead of “username@gmail.com“. This is way better if you want maximum portability of your identity, but want to keep the practicality of GMail’s integrated chat.
  • If you ever need to, you can upgrade some accounts to paid “Premier” accounts to get more space, reliability and support.

Cons

  • Not all Google services are available in Google Apps. For example, Google Reader or Picasa or Google Maps are not included in Google Apps, and you would end up having to login using your public GMail account. This means that whatever links you see at the top (“Mail”, “Calendar”, “Documents”, etc.) will take you to the wrong application (the public one instead of your Google Apps one). Also, your beloved iGoogle homepage will only work in the “public Google” space, so you won’t be able to make it work with your Google Apps things. Instead, in Google Apps, you’ll have the lame and ugly “Partner Homepage”.
  • As mentioned in the previous point, Google still hasn’t fixed the whole Google Accounts debacle. Namely, your regular public account will still be needed to log into non-Apps places like Google Reader, Picasa, Google Maps, etc. You can’t use your Google Apps account for this, which results in a slightly schizophrenic user experience. The cookies seem to use different tokens, though, so you can transparently be logged in to Google Apps and public Google services using the same browser.
  • Your chat history won’t be migrated, nor will be your chat buddies (for this you’ll have to re-invite people to chat with you at your new “username@domain.com” address).

The Procedure

If you’ve decided that Google Apps is still for you, then this is how you can migrate your regular Google account (GMail, GCal, etc.) to Google Apps.

First Steps

  1. Create a free Google Apps account.
    1. If the main (administrator) account you created is not going to be your actual migrated user account, create that account.
    2. In “Domain Settings”, check “Automatically add new Google services” and “Turn on new features”, so that you’re as close as possible to the actual GMail’s features. This is how you can get GMail Labs, which we’ll use for the migration. You may also want to check “Enable SSL”, for added security.
  2. Get a second browser, because we’re going to go back and forth between GMail and Google Apps. Sure, Google seems to now support being logged into both kinds of account at the same time, so you could do it with one browser without needing to login/logout all the time, but those pages look an awful lot like each other and if you’re using one browser it would be easy to get mixed up. With a second browser, you have an added difference that helps in not doing things in the wrong place. If you’re using a non-standard browser like Firefox or Opera, you can use the standard one (IE or Safari). If not, now’s a good time to get Firefox and join the cool kids. Alternatively, if you have 2 screens, you could use 2 browser windows on each one. That works too.

Migrate your email

  1. If you previously already had a Google Apps account and were forwarding your custom domain’s email to GMail, stop the forwarding now.
  2. Start forwarding your GMail to Google Apps. This will ensure that any email arriving during the migration will end up in Google Apps instead. You do that by going into “Settings” > “Forwarding and POP/IMAP”. Set it to delete the copy in GMail, as you don’t want to have duplicates of all your messages.
  3. Migrate your filters
    1. Enable the “Import/Export Filters” feature in GMail Labs. Go to “Settings” > “Filters”, select all your filters and click “Export”.
    2. Enable the “Import/Export Filters” in Google Apps Labs. Go to “Settings” > “Filters” and import the file you just created. This will create all the labels you’re using in your filters.
  4. Migrate your labels
    1. Create manually any label that wasn’t created using the filters. Note those down, because we’re going to have to reassign those labels manually too (the other labels will be assigned automatically by the filters as email messages are imported).
  5. Migrate your contacts
    1. Go to your GMail’s Contacts and click “Export”. Choose “Everyone (All Contacts)”, and “Google’s CSV format”.
    2. Go to your Google Apps’ Contacts and click “Import”. Choose the file you just created and check the results.
    3. Even though we’re using Google’s own CSV format, for some reason it messed up some of my contacts, those with many email addresses. They would end up with 2 or 3 email addresses all concatenated together with a semi-colon separator. I had to then cut the appended addresses and put them back in their own slot. This means some of your filters or searches may not work (if they’re trying to match against one of those messed up addresses) so you may have to go through your contact list and check your contacts. Also, see the note later in the “Migrate your messages” step.
    4. If you want to preserve your contacts groups, it’s a bit more painful. You need to export each group separately to a CSV file, and import it by choosing “Also add these new contacts to [new group…]”. Don’t worry, it will reuse already imported contacts – you won’t end up with duplicates. However, if you have lots of groups, it can take a while to do that. Also, the popup you get to fill the new group name is pretty crappy and will mess up any non-english character you enter. You can go back to those groups and fix their names later.
  6. Migrate your messages. For this, you have 2 options: the simple way, or the complex way.
    1. The simple way is to use GMail Backup. Just like the complex way below, GMail Backup (as of this writing) won’t migrate your Chat history. However, it should be as simple as pressing 2 buttons (one to backup your GMail account, one to restore it to your Google Apps’ inbox). I haven’t used it so I don’t know how well it works, or how long it takes, though.
    2. A slightly more complex version of the simple way (yeah, I know, it doesn’t make much sense) is to use the imapsync solution designed by Tyler Ham.
    3. The complex way uses Google’s own mail fetcher. There are a few valid reasons for it, but there are caveats too:
      1. Pros
        1. You won’t get slapped for uploading too much to your Google Apps account.
        2. Google will take care of everything, and you can turn off your computer in the meantime. This also means you don’t care about unexpected interruptions like your computer crashing, your internet connexion getting weird, or your ex-girlfriend setting your room on fire.
      2. Cons
        1. It’s called the “complex way” because you’ll need to sort some of your messages again. All messages pulled by the mail fetcher will be sorted automatically by your filters, but you may need to do some additional sorting for those messages that are exceptions. In my case, that was actually a good thing, as it forced me to make my filters more efficient, made me get rid of junk messages I somehow missed, and more generally cleaned up my account.
        2. It can take a week or so to transfer everything, depending on how many messages you have. Yes. A week. I myself had roughly 52000 emails taking 2.6Gb, and it took 6 days for Google to pull everything in.
        3. You won’t be able to access your GMail account through POP during this time (IMAP is fine, though).
      3. If you think you’re up to it here’s how it works:
        1. In your GMail account, go to “Settings” > “Forwarding and POP/IMAP”.
        2. Enable POP for all messages, even those already downloaded. At this point, if you were using POP access before, you may want to stop for a while, otherwise your tool may re-download all your messages over again. Now, if you use POP access for some reason, there’s a good chance you want to make it point to your Google Apps’ account now, so it shouldn’t be too problematic.
        3. In Google Apps’ mail, go to “Settings”, “Accounts” and add a POP account for your original GMail account. Google will start pulling all your messages to put them in your new Google Apps inbox, and you can monitor the progress by clicking on “View history”. Don’t worry if you see Google telling you there’s only, like, a couple hundred messages left in GMail, it’s just confused. It will discover new messages as it goes along. The main thing you should be looking for is whether it keeps pulling 200 messages every 30 minutes or so. Additionally, you may want to use one of the options, like archive all messages automatically, or label them all with some label of your choice.
        4. While your old email comes in, check the Spam folder every day. You’ll see that Google Apps flags some of that email as spam, and you’ll have to teach him that it’s not. Almost all the time, however, those false positives would be from contacts that got messed up during the “Migrate your contacts” step, so it’s another way of figuring out which ones need fixing.
        5. Unfortunately, your Chat history won’t get migrated this way. Maybe Google will one day expose Chats as an IMAP folder we can then migrate, or someone will find a way to get that from the Jabber server or something.
  7. Migrate your quick links, if you’re using that GMail Labs feature. You have to do that manually, unfortunately.
    1. Click on each quick link in your GMail account, copy the search box’s text.
    2. Go to your Google Apps’ inbox, paste the search query in the search box, and click “Create quick link”.

Migrate your documents

For some reason, Google refuses to change the ownership of documents from a “@gmail.com” user to a “@yourdomain.com” user, even if you configured your Google Apps domain so that users can share documents freely with other domains. So we have to move them by hand. Good thing somebody wrote a script for that already:

  1. Download all your documents to your disk.
  2. Re-upload them to your Google Apps’ account.
  3. You may have to reconfigure the sharing of your documents.

Migrate your calendars

  1. For your main calendar, in your regular Google Calendar, go to “Settings” > “Export Calendars” and save the file to disk. In Google Apps, import that file in “Settings” > “Import Calendar”.
  2. For the other calendars, do this:
    1. In your regular Google Calendar, go to “Settings”, “Calendars”, and click on each one of them.
    2. Look at the calendar address line and copy the Calendar ID.
    3. In Google Apps, paste that ID in the “add a friend’s calendar” text box.
  3. Some calendars require some kind of authentication (like your Facebook Events calendar), so you may have to go back to the source and get the calendar’s address again.
  4. If you previously had a Google Apps email address, go to Google Accounts and check that you didn’t add it as an alternate email address tied to your regular GMail account. This is how you might have gotten invitations to events sent to your own domain address in your Google Calendar. Now that you’re using the “real” Google Apps calendar, you don’t need this anymore.

Google Sync and other stuff

  1. If you have a smartphone and you’re using Google Sync, update the login parameters by replacing your old GMail address with the new custom domain one.
  2. If you have Firefox extensions that have your login information, update those as well.
  3. Same thing goes for any desktop notifier you may have.

Finish migrating your messages

  1. If you were using Google’s mail fetcher for migrating your messages, you need to do a few things after it’s done.
    1. When your email has been successfully migrated to your Google Apps inbox, setup Thunderbird with an IMAP access for both your old GMail account and your new Google Apps one. You can use another client if you want but that’s what I used.
    2. For each label that was not automatically re-applied through a filter, do the following:
      1. Create the label in Google Apps if it didn’t exist already.
      2. Go into that label’s folder in Thunderbird, in your old GMail account.
      3. Copy all the messages into the corresponding label folder in your Google Apps account. This won’t duplicate your messages. Instead, it will apply that label. This is a simple way to reapply labels to messages. However, the read/unread status of those messages may get lost.

Current bugs and issues

This is a list of the current big issues in the migration process.

  • Export and import of contacts, even using Google’s CSV format, is broken between public and Apps GMail. Contacts with a lot of email addresses (more than 3 or 4, apparently) have half their addresses ending up concatenated at destination.
  • Export and import of contacts doesn’t support contact groups information.
  • Importing a contact list into a new group opens a popup text box where the user enters the name of that new group. That text box messes up non-english characters.
  • No export/import of Quick Links.
  • No way to change ownership of documents from the “gmail.com” domain to a custom domain.

Labels and quick links work together

I just realized that the 2 GMail Labs experiments “Go to label” and “Quick links” work together, which makes quick links all the more useful. So say you have 2 labels named “Newsletters” and “Notifications”, and one quick link named “ALT.NET” (which finds all the posts from the ALT.NET mailing list). If you summon the “go to label” popup and start typing “n”, it will show all three:

Now, you ask, when should you use a label, and when should you use a quick link?

The main difference is that quick links are only search queries, whereas labels can be both search queries (a filter that automatically assigns that label) or manually maintained containers. If you need the latter, that’s a no brainer. If, however, what you want can be expressed with a search query, you need to choose between labels and quick links.

My method, so far, is based on 2 criteria: complexity and scalability.

  • Make a quick link if the search query is simple, and make a filter if it is complex. Typically, finding out all the emails from a specific mailing list is as simple as typing “list:altdotnet” in the search box, so this calls for a quick link. On the other hand, finding the receipts from some company that also sends you various newsletters and announcements can be a bit more complex, so I would make that a filter. The reason for this is that if the search query is complex, there’s a good chance it’s not complex enough. There’s a good chance some emails won’t get caught, because you didn’t think about all the cases or somehow an exceptional case shows up in your inbox. When this happens, you can still tag that message manually and keep going.
  • Make a quick link if similar types of containers exist. For example, “ALT.NET” finds all the mail from a specific mailing list. There could be dozen of similar containers if I’m subscribed to a dozen of other mailing lists. I don’t want to clutter my email organisation with dozens of such labels, so I go for a quick link. My labels tend to be generic concepts that won’t scale up much: “Newsletters”, “MailingLists”, “Receipts”, some labels for my different internet identities and/or email accounts, and some GTD-ish labels (“FollowUp”, “Hold”).

I hope this helps.