The Stochastic Game

Ramblings of General Geekery

Migrating to Google Apps

Migrating from a regular public Google account (GMail, etc.) to a Google Apps account seems to be a hot topic among geeks. Lots of people did it and posted their experience on their blog, which is often helpful for the next ones to try it. Since I recently migrated my account too, I thought I’d share this here. The important difference is that most people only post how they migrate their email. I tried to post about a lot more than that, including how to migrate contacts and groups and filters and quick links and documents and all that. I also wrote a complete “pros & cons” section up front so you can check whether Google Apps is for you.

Pros and Cons

Pros

  • You can administer your domain using Google’s control panel. All the users in your group (probably your family, or some kind of circle of friends) have a common ground to communicate on (chat integrated in mail, calendars, documents sharing, etc.). Scott Hanselman has a few pros & cons on that subject in his article about migrating his whole family to Google Apps, and the follow-up article . Note that these articles are outdated in a few places (some things have been fixed, some methods of backup/transfer don’t work anymore, some pros/cons are not valid anymore).
  • Your integrated GMail chat will now make you appear online as “username@domain.com” instead of “username@gmail.com“. This is way better if you want maximum portability of your identity, but want to keep the practicality of GMail’s integrated chat.
  • If you ever need to, you can upgrade some accounts to paid “Premier” accounts to get more space, reliability and support.

Cons

  • Not all Google services are available in Google Apps. For example, Google Reader or Picasa or Google Maps are not included in Google Apps, and you would end up having to login using your public GMail account. This means that whatever links you see at the top (“Mail”, “Calendar”, “Documents”, etc.) will take you to the wrong application (the public one instead of your Google Apps one). Also, your beloved iGoogle homepage will only work in the “public Google” space, so you won’t be able to make it work with your Google Apps things. Instead, in Google Apps, you’ll have the lame and ugly “Partner Homepage”.
  • As mentioned in the previous point, Google still hasn’t fixed the whole Google Accounts debacle. Namely, your regular public account will still be needed to log into non-Apps places like Google Reader, Picasa, Google Maps, etc. You can’t use your Google Apps account for this, which results in a slightly schizophrenic user experience. The cookies seem to use different tokens, though, so you can transparently be logged in to Google Apps and public Google services using the same browser.
  • Your chat history won’t be migrated, nor will be your chat buddies (for this you’ll have to re-invite people to chat with you at your new “username@domain.com” address).

The Procedure

If you’ve decided that Google Apps is still for you, then this is how you can migrate your regular Google account (GMail, GCal, etc.) to Google Apps.

First Steps

  1. Create a free Google Apps account.
    1. If the main (administrator) account you created is not going to be your actual migrated user account, create that account.
    2. In “Domain Settings”, check “Automatically add new Google services” and “Turn on new features”, so that you’re as close as possible to the actual GMail’s features. This is how you can get GMail Labs, which we’ll use for the migration. You may also want to check “Enable SSL”, for added security.
  2. Get a second browser, because we’re going to go back and forth between GMail and Google Apps. Sure, Google seems to now support being logged into both kinds of account at the same time, so you could do it with one browser without needing to login/logout all the time, but those pages look an awful lot like each other and if you’re using one browser it would be easy to get mixed up. With a second browser, you have an added difference that helps in not doing things in the wrong place. If you’re using a non-standard browser like Firefox or Opera, you can use the standard one (IE or Safari). If not, now’s a good time to get Firefox and join the cool kids. Alternatively, if you have 2 screens, you could use 2 browser windows on each one. That works too.

Migrate your email

  1. If you previously already had a Google Apps account and were forwarding your custom domain’s email to GMail, stop the forwarding now.
  2. Start forwarding your GMail to Google Apps. This will ensure that any email arriving during the migration will end up in Google Apps instead. You do that by going into “Settings” > “Forwarding and POP/IMAP”. Set it to delete the copy in GMail, as you don’t want to have duplicates of all your messages.
  3. Migrate your filters
    1. Enable the “Import/Export Filters” feature in GMail Labs. Go to “Settings” > “Filters”, select all your filters and click “Export”.
    2. Enable the “Import/Export Filters” in Google Apps Labs. Go to “Settings” > “Filters” and import the file you just created. This will create all the labels you’re using in your filters.
  4. Migrate your labels
    1. Create manually any label that wasn’t created using the filters. Note those down, because we’re going to have to reassign those labels manually too (the other labels will be assigned automatically by the filters as email messages are imported).
  5. Migrate your contacts
    1. Go to your GMail’s Contacts and click “Export”. Choose “Everyone (All Contacts)”, and “Google’s CSV format”.
    2. Go to your Google Apps’ Contacts and click “Import”. Choose the file you just created and check the results.
    3. Even though we’re using Google’s own CSV format, for some reason it messed up some of my contacts, those with many email addresses. They would end up with 2 or 3 email addresses all concatenated together with a semi-colon separator. I had to then cut the appended addresses and put them back in their own slot. This means some of your filters or searches may not work (if they’re trying to match against one of those messed up addresses) so you may have to go through your contact list and check your contacts. Also, see the note later in the “Migrate your messages” step.
    4. If you want to preserve your contacts groups, it’s a bit more painful. You need to export each group separately to a CSV file, and import it by choosing “Also add these new contacts to [new group…]”. Don’t worry, it will reuse already imported contacts – you won’t end up with duplicates. However, if you have lots of groups, it can take a while to do that. Also, the popup you get to fill the new group name is pretty crappy and will mess up any non-english character you enter. You can go back to those groups and fix their names later.
  6. Migrate your messages. For this, you have 2 options: the simple way, or the complex way.
    1. The simple way is to use GMail Backup. Just like the complex way below, GMail Backup (as of this writing) won’t migrate your Chat history. However, it should be as simple as pressing 2 buttons (one to backup your GMail account, one to restore it to your Google Apps’ inbox). I haven’t used it so I don’t know how well it works, or how long it takes, though.
    2. A slightly more complex version of the simple way (yeah, I know, it doesn’t make much sense) is to use the imapsync solution designed by Tyler Ham.
    3. The complex way uses Google’s own mail fetcher. There are a few valid reasons for it, but there are caveats too:
      1. Pros
        1. You won’t get slapped for uploading too much to your Google Apps account.
        2. Google will take care of everything, and you can turn off your computer in the meantime. This also means you don’t care about unexpected interruptions like your computer crashing, your internet connexion getting weird, or your ex-girlfriend setting your room on fire.
      2. Cons
        1. It’s called the “complex way” because you’ll need to sort some of your messages again. All messages pulled by the mail fetcher will be sorted automatically by your filters, but you may need to do some additional sorting for those messages that are exceptions. In my case, that was actually a good thing, as it forced me to make my filters more efficient, made me get rid of junk messages I somehow missed, and more generally cleaned up my account.
        2. It can take a week or so to transfer everything, depending on how many messages you have. Yes. A week. I myself had roughly 52000 emails taking 2.6Gb, and it took 6 days for Google to pull everything in.
        3. You won’t be able to access your GMail account through POP during this time (IMAP is fine, though).
      3. If you think you’re up to it here’s how it works:
        1. In your GMail account, go to “Settings” > “Forwarding and POP/IMAP”.
        2. Enable POP for all messages, even those already downloaded. At this point, if you were using POP access before, you may want to stop for a while, otherwise your tool may re-download all your messages over again. Now, if you use POP access for some reason, there’s a good chance you want to make it point to your Google Apps’ account now, so it shouldn’t be too problematic.
        3. In Google Apps’ mail, go to “Settings”, “Accounts” and add a POP account for your original GMail account. Google will start pulling all your messages to put them in your new Google Apps inbox, and you can monitor the progress by clicking on “View history”. Don’t worry if you see Google telling you there’s only, like, a couple hundred messages left in GMail, it’s just confused. It will discover new messages as it goes along. The main thing you should be looking for is whether it keeps pulling 200 messages every 30 minutes or so. Additionally, you may want to use one of the options, like archive all messages automatically, or label them all with some label of your choice.
        4. While your old email comes in, check the Spam folder every day. You’ll see that Google Apps flags some of that email as spam, and you’ll have to teach him that it’s not. Almost all the time, however, those false positives would be from contacts that got messed up during the “Migrate your contacts” step, so it’s another way of figuring out which ones need fixing.
        5. Unfortunately, your Chat history won’t get migrated this way. Maybe Google will one day expose Chats as an IMAP folder we can then migrate, or someone will find a way to get that from the Jabber server or something.
  7. Migrate your quick links, if you’re using that GMail Labs feature. You have to do that manually, unfortunately.
    1. Click on each quick link in your GMail account, copy the search box’s text.
    2. Go to your Google Apps’ inbox, paste the search query in the search box, and click “Create quick link”.

Migrate your documents

For some reason, Google refuses to change the ownership of documents from a “@gmail.com” user to a “@yourdomain.com” user, even if you configured your Google Apps domain so that users can share documents freely with other domains. So we have to move them by hand. Good thing somebody wrote a script for that already:

  1. Download all your documents to your disk.
  2. Re-upload them to your Google Apps’ account.
  3. You may have to reconfigure the sharing of your documents.

Migrate your calendars

  1. For your main calendar, in your regular Google Calendar, go to “Settings” > “Export Calendars” and save the file to disk. In Google Apps, import that file in “Settings” > “Import Calendar”.
  2. For the other calendars, do this:
    1. In your regular Google Calendar, go to “Settings”, “Calendars”, and click on each one of them.
    2. Look at the calendar address line and copy the Calendar ID.
    3. In Google Apps, paste that ID in the “add a friend’s calendar” text box.
  3. Some calendars require some kind of authentication (like your Facebook Events calendar), so you may have to go back to the source and get the calendar’s address again.
  4. If you previously had a Google Apps email address, go to Google Accounts and check that you didn’t add it as an alternate email address tied to your regular GMail account. This is how you might have gotten invitations to events sent to your own domain address in your Google Calendar. Now that you’re using the “real” Google Apps calendar, you don’t need this anymore.

Google Sync and other stuff

  1. If you have a smartphone and you’re using Google Sync, update the login parameters by replacing your old GMail address with the new custom domain one.
  2. If you have Firefox extensions that have your login information, update those as well.
  3. Same thing goes for any desktop notifier you may have.

Finish migrating your messages

  1. If you were using Google’s mail fetcher for migrating your messages, you need to do a few things after it’s done.
    1. When your email has been successfully migrated to your Google Apps inbox, setup Thunderbird with an IMAP access for both your old GMail account and your new Google Apps one. You can use another client if you want but that’s what I used.
    2. For each label that was not automatically re-applied through a filter, do the following:
      1. Create the label in Google Apps if it didn’t exist already.
      2. Go into that label’s folder in Thunderbird, in your old GMail account.
      3. Copy all the messages into the corresponding label folder in your Google Apps account. This won’t duplicate your messages. Instead, it will apply that label. This is a simple way to reapply labels to messages. However, the read/unread status of those messages may get lost.

Current bugs and issues

This is a list of the current big issues in the migration process.

  • Export and import of contacts, even using Google’s CSV format, is broken between public and Apps GMail. Contacts with a lot of email addresses (more than 3 or 4, apparently) have half their addresses ending up concatenated at destination.
  • Export and import of contacts doesn’t support contact groups information.
  • Importing a contact list into a new group opens a popup text box where the user enters the name of that new group. That text box messes up non-english characters.
  • No export/import of Quick Links.
  • No way to change ownership of documents from the “gmail.com” domain to a custom domain.

Labels and quick links work together

I just realized that the 2 GMail Labs experiments “Go to label” and “Quick links” work together, which makes quick links all the more useful. So say you have 2 labels named “Newsletters” and “Notifications”, and one quick link named “ALT.NET” (which finds all the posts from the ALT.NET mailing list). If you summon the “go to label” popup and start typing “n”, it will show all three:

Now, you ask, when should you use a label, and when should you use a quick link?

The main difference is that quick links are only search queries, whereas labels can be both search queries (a filter that automatically assigns that label) or manually maintained containers. If you need the latter, that’s a no brainer. If, however, what you want can be expressed with a search query, you need to choose between labels and quick links.

My method, so far, is based on 2 criteria: complexity and scalability.

  • Make a quick link if the search query is simple, and make a filter if it is complex. Typically, finding out all the emails from a specific mailing list is as simple as typing “list:altdotnet” in the search box, so this calls for a quick link. On the other hand, finding the receipts from some company that also sends you various newsletters and announcements can be a bit more complex, so I would make that a filter. The reason for this is that if the search query is complex, there’s a good chance it’s not complex enough. There’s a good chance some emails won’t get caught, because you didn’t think about all the cases or somehow an exceptional case shows up in your inbox. When this happens, you can still tag that message manually and keep going.
  • Make a quick link if similar types of containers exist. For example, “ALT.NET” finds all the mail from a specific mailing list. There could be dozen of similar containers if I’m subscribed to a dozen of other mailing lists. I don’t want to clutter my email organisation with dozens of such labels, so I go for a quick link. My labels tend to be generic concepts that won’t scale up much: “Newsletters”, “MailingLists”, “Receipts”, some labels for my different internet identities and/or email accounts, and some GTD-ish labels (“FollowUp”, “Hold”).

I hope this helps.


Sync multiple Google Calendars with your iPhone

If you have an iPhone, or any other smartphone for that matter, you will probably by now have set up Google Sync with it so you can synchronize your contacts and calendar.

However, if you’re a bit hasty and follow the simple tutorial that Google provides, you will end up synchronizing  only your main calendar with your phone. It’s easy to miss the fact you can actually synchronize up to 5 (at the time of this writing) calendars. If you go back to Google Sync from your device, you should then be able to select which calendars you wish to synced. This is especially useful if you want to sync your Remember The Milk tasks or your Facebook events.

Note

To get Remember The Milk tasks in your calendar, go here.

To get Facebook events in your calendar, go to your events page, and click “Export Events”, at the top of the page. You’ll get an URL that you can then import in Google Calendar or any other calendar application.


Conditional operator fun

A good friend of mine sent me this little “gotcha” a few days ago, so I thought I’d share:

#include <iostream>

#include <string>
void main() {
int a = 0;
int* ptrA = &a;
const int b = 1 ;

const int& refA1 = (ptrA ? *ptrA : 1);
const int& refA2 = (ptrA ? *ptrA : b);

std::cout << refA1 << std::endl;
std::cout << refA2 << std::endl;

a = 42;

std::cout << refA1 << std::endl;
std::cout << refA2 << std::endl;
}

This prints:

0

0

0

0

Turns out I didn’t really know how the “?” operator figures out its overall return type because most of the time I use it with class pointers or strings or something not as complicated as integers with values of 0 or 1 or even (crazy!) 42.

In this case, the “?” operator looks at an “int” and a “const int” and figures, hey, I should really make all this a “const int”! And what better way to make a “const int” out of an “int”? Well, copy it somewhere safe, of course! So you end up with constant copies of “a”. You can print out the addresses of all the variables to see how “refA1” and “refA2” point to new places.

If you remove the “const” from “b”’s declaration, the “?” operator then looks at two integers, makes “refA2” reference “a”’s address directly, and you end up with the following output:

0
0
0
42

Conclusion: watch out when initializing a variable that could either be a constant (earth’s gravity) or a user tweakable value (gravity on the space station in level 3 of your game).


You know you use PowerShell too much when…

…you go back to Bash, and try to type this:

ls –R | grep “foo”

…and wonder why it’s not finding any file with “foo” in it.


Pass-phrases, and the problem with dumb websites

I’m a huge fan on pass-phrases. Since Jeff already evangelised them over passwords, giving arguments and advice, there’s no need to add anything…

Except, well, some good old complaining.

It pisses me off that some websites have a limited length for passwords, thus preventing users from using pass-phrases. It’s not a pass-phrase if it can only have a maximum of 12 characters, is it? But the worst is how most of those websites won’t even warn you that your password is too long… they will just truncate it and tell you everything’s okay! And then, the next time you log in, you spend 10 minutes wondering how you can mistype 50 times a passphrase you’re absolutely sure about.

Now, time to point some fingers. Recently, the 2 websites that gave me this kind of crappy user experience were the Archos Store (after creating a new account) and Linked In (after changing my password on an existing account). It’s especially surprising how a high profile website like Linked In can be so poorly implemented.


Disable windows from resuming when opening your laptop’s lid

One of my laptops is getting old and the lid is not as sturdy as it used to be. It now has the unwanted tendency of triggering a “laptop lid open” event when you barely touch it because the lid moves up a bit and back down. This is problematic because it wakes up the operating system, which doesn’t always detect that the lid was closed immediately.

When you run an internet search about laptop lids and putting Windows on stand by or hibernate, you find a lot of stuff, but nothing useful about disabling resume. To solve this problem you need to think like a programmer, i.e. find a solution that kinda makes sense, but not really.

Obviously, the answer lies in the Power Options dialog, but the only setting you have access to is what to do when the lid is closed, not when it is open.

Well, here comes the shocking answer: if you put your computer on stand by when you close your lid, it will wake up when you open it. But if you don’t do anything when you close your lid, it won’t do anything either when you open it!

Setting that first combobox above to “Do nothing” therefore fixed my problem. Now, I can close the lid, let the computer go to sleep after 5 minutes of inactivity, and move it around without fear of it waking up because the lid is too sensitive. I do have to press the power button to wake it up instead of just opening it, but that’s not too awful.


Visual Studio Tips & Tricks

Stephen Walther recently blogged about tricks that every developer should know about Visual Studio. Most of those he mentioned I use on a daily basis, so I highly recommend them. It actually surprises me how many other programmers don’t know about those features. Somehow, most programmers know Visual Studio as much as they know Microsoft Word: there’s a big space in the middle to type your text, and then maybe they know a couple of menus and shortcuts and that’s it.

Anyway, here are a few other features I use frequently:

Tip #1 – Use CTRL-I for incremental search

I’m a big fan of incremental search. It’s actually the reason I originally switch from Internet Explorer to Firefox, back when Firefox was in beta (I already had tabs with Netcaptor). By the way, I’m happy to see IE8 now also features incremental search but, well, I’m hooked on FF now… anyway…

In Visual Studio, the incremental search mode is entered with CTRL-I. Then you can start typing right away what you want to find. You can also use backspace and all, as expected. The only problem is that it uses the last search mode defined in the “Find” dialog, so if the “Match case” option is currently checked, it will only perform incremental search on what you type in a case sensitive way. This may be a bit confusing at first, but using incremental search really makes navigating code faster.

Tip #2 – Use navigate backward and forward

Like the first tip, this tip make it faster to navigate code (because we’re spending more time reading code than writing code).

CTRL– (minus sign) and CTRL-SHIFT– allow you to respectively navigate backward and forward. The first one is the one I use almost all the time: I’m looking at some code, see a call to a function, go to the definition of that function, search for something, and then I want to go back to where I was at first. Well, without having to worry about anything, I can navigate backward twice and I end up where I want. No need to find the correct tab, figure out if I stayed in the same file or not while jumping through the code, etc. Wicked!

Tip #3 – Define some external tools

You can define “external tools” with, surprisingly, the Tools > External Tools menu item. A common thing for me is to define revision control related stuff there because most of the time I don’t like how buggy and slow source control plugins are. I also do this in Visual Studio Express where there is no support for plugins at all in the first place.

For example, say you’re using Perforce. You can create a new external tool like this:

It will checkout the current file (the one you’re editing if the focus is in the text editor, or the one selected in the Solution Explorer if that’s where the focus is). Now you just need to bind “Tools.ExternalCommand2” (because it’s the second external tool) to the keyboard shortcut of you choice (mine is CTRL-K, CTRL-E for “open for edit”). Shazam! Just use your keyboard shortcut and you can checkout a file without leaving the text editor.

You can setup other commands, like show the history of the file or make a diff with the previous version. You can also easily adapt this to other systems, like SubVersion (except in this case you won’t need the “open for edit” command because SVN doesn’t lock files by default).