Ramblings of General Geekery

Poor man’s search engines sync for Google Chrome

Update: since Lifehacker featured this post on their home page, I released some simpler updated version of the scripts here.

The wonderful thing about open-source software is that whenever something’s missing, anybody can go ahead and fix it… that is, unless nobody cares enough about it. And that’s precisely what’s happening with the “search engines sync” feature in Google Chrome, which has been in Chromium’s bug database for more than a year and a half. Looks like Google is not so much in a hurry to let you use other search engines as soon as you install a fresh copy of their browser.

Oh, don’t look at me. I seriously don’t have the time to dive into such a big codebase to add the feature myself… but what I do have the time for is a little scripted hack to get around the problem quickly!

Google Chrome, like a lot of programs these days, is using SQLite to store its data. Your search engines are stored in a database called “Web Data”, in a table called “keywords”. It’s quite easy to use the SQLite command line shell to export and import that table across your computers… so that’s what I did. There are 2 scripts: one to export, and one to import, although I called them “push” and “pull” to sound cool like the Git/Mercurial guys.

Windows

On Windows, my scripts are written in Powershell, which ships by default with Windows 7. On previous versions of Windows, Powershell ships with some service packs, so if you’re up to date you should have it. Otherwise, go get it from Microsoft’s website.

Update: if you’ve never run any Powershell script before, you’ll have to change the security settings because I didn’t sign my scripts. Run Powershell as an administrator and type “Set-ExecutionPolicy RemoteSigned”.

You’ll need to also download the SQLite command line shell (no install required, it’s just one .exe file in a zip). Just keep it next to the script files.

Here’s the export script:

param (
    [String] $Destination = "keywords.sql"
)

$CurrentDir = Split-Path -Parent $MyInvocation.MyCommand.Path
if (![IO.Path]::IsPathRooted($Destination)) { $Destination = [IO.Path]::Combine($CurrentDir, $Destination) }
$Destination = $Destination.Replace('', '/')
$TempSqlScript = "$env:TEMPsync_chrome_sql_script"

Push-Location
Write-Output "Exporting Chrome keywords to $Destination..."
cd "$HOMEAppDataLocalGoogleChromeUser DataDefault"
Write-Output ".output `"$Destination`"" | Out-File $TempSqlScript -Encoding ASCII
Write-Output ".dump keywords" | Out-File $TempSqlScript -Encoding ASCII -Append
& "$CurrentDirsqlite3.exe" -init $TempSqlScript "Web Data" .exit
Remove-Item $TempSqlScript
Pop-Location

And here’s the import script:

param (
    [String] $Source = "keywords.sql"
)

if ((Get-Process -Name chrome -ErrorAction SilentlyContinue | Measure-Object).Count -gt 0)
{
    throw "Close Chrome and try again..."
}

$Reply = Read-Host -Prompt "This will overwrite your Google Chrome search engines! Are you sure?  "
if (!($Reply -match "^(Y|y|YES|Yes|yes)$"))
{
    Write-Output "Cancelling operation."
    exit
}

$CurrentDir = Split-Path -Parent $MyInvocation.MyCommand.Path
if (![IO.Path]::IsPathRooted($Source)) { $Source = [IO.Path]::Combine($CurrentDir, $Source) }
$Source = $Source.Replace('', '/')
$TempSqlScript = "$env:TEMPsync_chrome_sql_script"

Push-Location
Write-Output "Importing Chrome keywords from $Source..."
cd "$HOMEAppDataLocalGoogleChromeUser DataDefault"
Write-Output "DROP TABLE IF EXISTS keywords;" | Out-File $TempSqlScript -Encoding ASCII
Write-Output ".read `"$Source`"" | Out-File $TempSqlScript -Encoding ASCII -Append
& "$CurrentDirsqlite3.exe" -init $TempSqlScript "Web Data" .exit
Remove-Item $TempSqlScript
Pop-Location

MacOS X

On MacOS X I’m using bash so all you need is to make sure you’ve got the SQLite command line shell. You can either download it directly, or use one of the package managers like MacPorts or Homebrew (I’m using the latter myself).

Here’s the export script:

#!/bin/sh

DESTINATION=${1:-./keywords.sql}
TEMP_SQL_SCRIPT=/tmp/sync_chrome_sql_script
echo "Exporting Chrome keywords to $DESTINATION..."
cd ~/Library/Application Support/Google/Chrome/Default
echo .output $DESTINATION > $TEMP_SQL_SCRIPT
echo .dump keywords >> $TEMP_SQL_SCRIPT
sqlite3 -init $TEMP_SQL_SCRIPT Web Data .exit
rm $TEMP_SQL_SCRIPT

And here’s the import script:

#!/bin/sh
if ps -x | grep -v grep | grep Google Chrome > /dev/null; then
    echo "Close Chrome and try again..."
    exit 1
fi

read -p "This will overwrite your Google Chrome search engines! Are you sure?  " -n 1
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
    echo "Cancelling operation."
    exit 1
fi

SOURCE=${1:-./keywords.sql}
TEMP_SQL_SCRIPT=/tmp/sync_chrome_sql_script
echo
echo "Importing Chrome keywords from $SOURCE..."
cd ~/Library/Application Support/Google/Chrome/Default
echo DROP TABLE IF EXISTS keywords; > $TEMP_SQL_SCRIPT
echo .read $SOURCE >> $TEMP_SQL_SCRIPT
sqlite3 -init $TEMP_SQL_SCRIPT Web Data .exit
rm $TEMP_SQL_SCRIPT

Bringing it all together with Dropbox

Those scripts handle importing and exporting our search engines, but we’re missing the “sync” part so far. That’s where popular syncing service Dropbox comes into play (but you can do this with any other syncing service). Just put all this stuff (the SQLite command line shell, the scripts, and the database dump) in a Dropbox folder. All you have to do is run the “export/push” script after you’ve changed your search engines, and run the “import/pull” script next time you get on another computer. Dropbox will have synced the dumped database file (“keywords.sql” by default) in the meantime (unless you’re moving really super fast, in which case you’ll have to wait a second or two).

I’ve made a handy ZIP file with those scripts, a couple of bootstrap (double-clickeable) files for Windows, and the SQLite win32 command line shell. So download it and start syncing in a very hacky and lame way!


Using Twitter as a feed reader

Robert Scoble famously posted about ditching Google Reader for Twitter a bit more than a year ago, and ever since I’ve been baffled at people moving to Twitter or Facebook to read their news. Business Insider even said a few months ago that Twitter has killed RSS readers. I’ve always been wondering how Twitter would be any better than RSS… that is, until I’ve actually tried it.

Now I’m just thinking those people are crazy out of their minds.

First, let’s put Scoble aside. This guy is clearly not your average user, he’s really out there. I mean, his performance concerns with Google Reader are completely legitimate, but he must be the only guy on earth that actually follows more than 20,000 people… and still uses the term “friends” to define them, by the way. And I’m not even talking about the feasibility of keeping up with that much information — after all, if he’s clever, he’s using tools to consume it all and make the trending subjects bubble up to the surface, or maybe he does it the other way around by searching for topics he’s interesting in right now instead of reading the timeline in a linear fashion. But average people (or, well, at least me) are following real friends (either their blogs, or the links/videos/pictures they share), a few websites pertaining to their interests that may not be related to web technology (knitting? cat breeding?), and other things that can’t be processed by some “trending” tool.

Anyway, I used Twitter instead of Google Reader for a couple months, and here’s what I found.

User Experience

The first thing that bugs me about Twitter, and all of its client applications, is that it effectively has a worse user experience than even the crappiest of RSS feed reader.

Remember why RSS feeds were supposed to improve our productivity, making it easier to get to the information that mattered? They were supposed to do this by bringing the information to us instead of forcing us to go to the information. RSS feeds would deliver the content, and only the content, right there in front of us, and they pretty much delivered on that promise. Sure, some websites only publish an abstract of the article, and you need to go to that website to read the rest, but that’s the worst case scenario. The best case scenario is that you read the whole thing in a distraction free environment.

Now what about Twitter? The best case scenario is the title of the article, with a link to the website. Yep. As far as I’m concerned, this is a step backwards in terms of productivity: I read some entry on Twitter, click a shortened URL, wait for some HTTP redirections, and then end up on the full website with animated ads and other distractions. Compare that to reading the article right now with only the content in front of me. And I’m not even talking about the worst case scenario, which is a shortened title of the article with a link, or some obscure smart-ass comment with a link.

Then there’s the problem of knowing what you’ve already read. All RSS clients track the items you’ve read. Most even synchronize that data over the cloud. Compare that to Twitter which doesn’t remember anything and for which I always have to look out for that point in the timeline where I’ve caught up with all the new stuff. And that’s for a naive, linear approach. Most of the time, I’m going to read stuff in a non-linear way, either by using lists or filtering by user (on Twitter) or filtering by tag or website (on Google Reader), so the stuff I’ve already read could end up mixed up with the stuff I haven’t read yet.

Q&A

There’s one thing I keep reading about Twitter being awesome, and that’s its ability to give you answers super quickly. Supposedly, you can tweet a question and get replies within a few minutes, if not seconds.

Well, that’s the biggest load of crap I’ve ever heard.

First, this obviously works only if you’ve got a sizeable following. A year and a half ago, the average Twitter user had 127 followers. It’s hard to tell how that number was calculated, but let’s assume it did take into account statistical distortions from the few internet celebrities having insane numbers of followers on one end, and all the inactive users (supposedly making up 3/4 of the whole user base) on the other end. Either way, that number is just big enough for a user to get a few answers a day. To really get to the point where you get better answers faster than your search engine of choice, I’m guessing you need to hit the 500 followers mark, maybe even 1000. I wouldn’t know, I’m far from being that popular… And even then, what? Twitter gets better for you as you get more people to follow you? Ok… what else sounds like this? Yeah: pyramid schemes. And that’s known to be non sustainable. There will be a lots of users down there with 20 followers trying to get some attention.

Second, what happened to all that concern about privacy? People start complaining the second Google introduces their Web Search History, but on the other hand it’s OK to search for information by posting stuff publicly on Twitter, which is then indexed a hundred times more?

Technical Stuff

Then there’s the problem of infrastructure. This part has a lot more to do with personal opinions about stuff you may not care about, but I always prefer a more open, distributed design over a more closed, constrained one. For example, if I can sign into a website using either my Facebook or Google account or OpenID, I’ll take OpenID because it’s obviously more open and doesn’t rely on the existence of a single company. Sure, unless you’re running your own OpenID identity server, you’re relying on some third party provider, but you can abstract that from OpenID consumers, and then switch the underlying implementation anytime you want. To pick a more mainstream example, between using a Hotmail or Gmail address and a custom domain address, I’ll choose the custom domain address for similar reasons.

Ok, so this is on the same level as people using open-source software vs .people who don’t care, or people eating organic food vs. people who also don’t care. And in this case, you probably don’t care, and that’s fine, but I personally dislike how Twitter and other similar services create this “siloization” effect on information and data on the web.

On Using Twitter

All of this makes Twitter a pretty horrible experience as a feed reader, at least for me. Does that mean it sucks? Well, kinda, but not necessarily. First, some of the new features indicate how the design team is trying to reduce my first productivity problem: if a link in a tweet points to a picture or a video from a known media service, said picture or video will be displayed in the sidebar right away. When you couple this with the new keyboard shortcuts and the lists, you have an experience that’s somewhat close to that of Google Reader, at least for some of the tweets. I can easily imagine that it’s going to get closer and closer in the near future… but right now it’s still lagging far behind IMHO.

For now I’m back to using Twitter the way it’s effective for me: following actual people’s status updates and link shares. Twitter is wonderful in the way it lets you get closer to what experts and other key people in society have to say on an everyday basis (you wouldn’t believe how “I had pasta today” inspires you when it’s been tweeted by a Nobel prize winner…). Arguably, that’s actually what Twitter is all about: information through people. RSS on the other hand is only for delivering content (although Google is trying to add social layers on top of it). The two may feel similar, but not quite.

Twitter is also a useful alternative search engine in some situations, but more and more search engines actually include tweets as search results (in Google, you need to click on the “Realtime” result type on the left).

But all things being equal, if I get the choice between following the same thing on Twitter or via RSS, I’ll choose RSS for sure.