I love files. I love renaming them, moving them, sorting them, changing how they’re displayed in a folder, backing them up, uploading them to the internet, restoring them, copying them, and hey, even defragging them. As a metaphor for a way of storing a piece of information, I think they’re great. I like the file as a unit of work. If I need to write an article, it goes in a file. If I need to produce an image, it’s in a file.
I also love files. That’s why I put them in a NAS. And then I have another NAS to backup the first NAS. That’s why my blog is essentially generated from a bunch of files, and why my notes are in a wiki that’s also generated from a bunch of files. Files are easy to edit, store, version, backup, recover. Sure, they don’t scale very well to giant online services, but for personal use, they’re the absolute best in my opinion. Of course, they’re also the worst for companies who want to lock you in and profit from your data under cover of “convenience”.
But Apple doesn’t make it that easy to get to your files. Images are dumped into a big stream, sorted by date. Audio is somewhere in iTunes. Notes are… in a list? Apps are scattered around the desktop. Some of my files are in iCloud. You can email photos to liberate them from the iPhone, and through a convoluted method via iTunes, you can access some files within certain apps. But those files are transitory — cached, and may be deleted without warning. These aren’t like the carefully crafted files and folders on my computer.
This one of the reasons I have a love/hate relationship with my iOS devices. And if you look at the number of apps that support some Dropbox or Box integration or similar, I don’t think I’m the only one. I’m actually not even sure Dropbox would have been as popular if Apple had initially released iOS with friendly support for moving files in and out of the device. Most of the apps I use on a regular basis on my iPad are ones that can interact with the files in my home network, like Plex or GoodReader. I refuse to store stuff on iCloud (except as a form of extra backup), regardless of how many times Apple insists on running advertisements on my phone in the form of notifications you can’t disable1.
Anyway, read the whole article, as it’s making several other interesting points on the subject.
Bottom-line: I also miss files. We eventually got the big companies to sell music as DRM-free files, but it looks like it’s never going to happen for video. Everything is a cloud subscription service, now. Everything is streaming. Everything is a timeline. It’s not just about files going extinct, it’s about the very concept of “ownership” going obsolete.
Apple has been getting worse on that front, lately. ↩︎
I recently published version 0.8 of Wikked, my plain-text-files/SCM-backed
wiki engine, followed immediately by a quick little patch release. Grab it as
usual with a pip install wikked -U!
Although there are a few interesting new things in this release, the more
important announcement here is the launch of the official website
running itself on Wikked! Head over there now and tell me if anything
looks broken!
Keep reading if you want more information about all this.
New and Improved
There is now a simple way to upload files to your wiki – both as global files
(uploaded to the _files directory) or as page-local files (uploaded next to
the current page’s file). It looks like this and yes there is room for
improvement 🙂
…which leads me to the new help pages. You can see them on the official
website, but they come bundled with any Wikked wiki, since they’re part
of the source code. As such, they’re read-only (although you can edit them and
submit a pull request if you spot a mistake!). You can access them from the
“Help” link at the bottom of any wiki page.
Another thing you might have noticed with that link to the help section on
permissions is that Wikked now generates links for each heading on a page.
Last but not least, I re-did the UX of the navigation menu so that it finally
works correctly on both desktop and mobile browsers.
The Official Wiki
The other big new thing is the official Wikked wiki, which runs on
Wikked (unlike the previous official project page which was just a static page
generated with my other big web project, PieCrust).
Right now it looks a bit like this:
There’s not much to say about this besides “it’s about time”. There’s not even
much on that wiki yet – most of the content comes from the aforementioned new help pages.
I wonder how long it will take until somebody finds a security hole in Wikked
and adds a dick pic to the home page. Oh well, I’ll worry about that if Wikked
actually gets popular… and even then, if you really want to post dick pics,
you can already do that on the Wikked sandbox wiki or, well, anywhere
on the internet, really.
And that’s it! If you haven’t checked out Wikked yet, give it a go:
I originally wrote Wikked as a single page app because, well, I wanted to have a little fun and learn something new. And it worked (I did have fun learning Javascript past the “copy JQuery snippets from the internet” phase), but it didn’t really work (it made testing and deploying unnecessarily harder). It could have also been argued that making this an SPA was overkill… at least, it did bother me a bit, since I tend to prefer simpler/old-school technologies.
Anyway, I embarked on a giant refactor of the code base to reimplement the front-end using a more classic architecture, and the last couple versions of Wikked are the result of this transition. You shouldn’t see anything new, besides a few less old bugs, a few new bugs, and an overall more stable experience.
New documentation
In the process, I also got rid of the old documentation page I had on BOLT80, and made a proper versioned documentation website, just like with PieCrust. That website is part of the Wikked codebase so you can easily send patches for it… and of course, it’s made with PieCrust.
You can check it out at the same place as before on BOLT80. It’s still a work in progress design-wise – it’s meant to look similar to a Wikked-powered wiki, but it still lacks a bit of graphics and other stuff to make it look less sterile.
Next steps
As always after I spend a few months doing some giant task with the limited free time that I have, I need to get back to my other projects and reply to the various bug reports and pull requests I haven’t replied to in weeks. Then it’s back to business as usual, I hope… at least unless I get into another big refactor.
The big question I wanted to answer was how far you can go with just the default configuration, which is based on SQLite and requires no setup from the user. The reason for this was twofold:
I needed to write some advice in the documentation about when you should start looking into more sophisticated setups.
I plan to setup a public test wiki where people can try Wikked directly, and I needed to know if it would go down after I post the link on Reddit or HackerNews.
Initial assessment
The first thing I did was to figure out the current status of the code. For this, I took the first stress-test service I could find (which was Load Impact), and got my own private wiki tested.
This private wiki runs on the same server as this blog, which is a fairly under-powered server since almost all of my public websites are just static files, thanks to PieCrust: it’s a Linode VPS with only 512Mb of RAM.
The test requests a dozen different pages from the website, continually for around 10 seconds, with only a fraction of a second between each request. It increases the number of “users” running that test over time.
Here are some of the results:
As you can see, as the number of concurrent users increases, loading a page stays on average under a second, at 800ms. Then, around 20 concurrent users, things break down horribly and it can take between 3 and 10 seconds to load a page.
For a website running with SQLite on a server so small that Linode doesn’t even offer it anymore2, and designed mainly for private use, I think it’s pretty good. I mean, I initially didn’t plan for Wikked to run for groups larger than 10 or 15 people, let alone 20 people at the same time!
Still, I can obviously do better.
Request profiling
Werkzeug supports easy profiling of requests, so I added an option for that and looked at the output in QCacheGrind3. As I thought, pretty much all the time is spent running the SQL query to get the cached page, so there’s little opportunity to optimize the overall application’s Python code.
In Wikked, SQL queries are done through SQLAlchemy. This is because even though those queries are simple enough that even I could write them by hand, there are subtle differences in SQL dialects depending on the database implementation, especially when it comes to schema creation. I figured I would bypass the ORM layer if I need to in the future.
SQLAlchemy can be forced to log all SQL queries it generates, and that highlighted many simple problems. I won’t go into details but it boiled down to:
A couple of unnecessary extra queries, which came from my object model lazily loading stuff from the database when it didn’t need to.
Loading more columns than needed for the most common use-case of reading a page. Some of them would generate JOIN statements, too.
I also realized I was doing my main query against an un-indexed column, so I changed the schema accordingly… derp duh derp (I’m a n00b at this stuff).
Funkload
Now I was ready to run some more stress tests and see if those optimizations made a difference. But although Load Impact is a very cool service, it’s also a commercial service and I was running out of free tests. I didn’t want to spend money on this, since this is all just hobby stuff, so I looked for an alternative I could setup myself.
I found a pretty neat library called FunkLoad, which does functional and load testing. Perfect!
I started 4 Amazon EC2 instances, wrote an equivalent test script, and ran the test. To make it work, I had to install FunkLoad from source (as opposed to from pip), and troubleshoot some problems, but it worked OK in the end.
Without my optimizations, I got slightly better average page loads than before – probably coming from the fact that both my EC2 instances and my Linode server were on the west coast, whereas Load Impact was running from the east coast.
With the optimizations, however, it looked a lot better:
As you can see, Wikked on my small server can now serve 40 concurrent users without breaking a sweat: 300ms on average, and always less than 1s. And it could probably handle up to 50 or 60 concurrent users if you extrapolate the data a bit.
Moar hardware!
Next, I figured I would try to see if it made any difference to run the same setup (Wikked on SQLite) on a beefier server. I launched an EC2 instance that’s way better than my Linode VPS, with 3Gb of RAM and 2 vCPUs.
Well: yes, it does make a difference. This bigger server can serve 80 concurrent users while staying under the 1 second mark most of the time. Yay!
Conclusion
Those numbers may not seem like much but this is as good a time as any to remind you that:
I’m sticking to sub-1s times as the limit, because I like fast websites. But I could easily move the limit up to 1.5 seconds and still be within a generally acceptable range (e.g. from my home laptop, Wikipedia serves its pages in around 1.3 seconds).
This is about testing the most simple Wikked setup, based on SQLite, because that means the easiest install experience ever compared to other wikis that need a proper SQL server. And SQLite is notoriously limited in terms of concurrent access.
Serving even just 40 concurrent users is actually quite high. If you consider, say, 10 minutes per visit on average, that’s around 240 visitors per hour, or 1920 visitors per day if they’re all going to be mostly coming from the same time zone. That’s more than 50.000 visitors a month4.
Still, this is my first real web application, so there’s probably even more room for improvement. I’m always open to suggestions and constructive criticism, so check-out the code and see if you can spot anything stupid!
In the meantime, I’ve got some documentation to update, and a public test wiki to setup!
It’s still missing a custom theme and a fancy logo, by the way. That will be coming as soon as I have any actual idea of what to do there! ↩︎
It’s not a typo. QCacheGrind is a Qt version of KCacheGrind, so that you don’t need to install KDE libraries, and it looks slightly less terrible. ↩︎
The real issue is however how your site will behave if all of a sudden a lot of those visitors arrive at the same time. This is probably not uncommon if you have the kind of wiki where there can be announcements posted to a mailing list or a Facebook group, which can in turn get a lot of members to click the same link. ↩︎
There hasn’t been any updates on this blog for a few months, and there was a good reason for that: I was working on someting new.
The problem is that I was trying to get this new project to a “good enough” state to launch publicly… but somehow I ended up in a seemingly infinite loop of improvements, refactorings, and bug fixing.
Eventually I snapped out of it: fuck it, let’s launch it as is, and see if anybody cares enough to complain that it’s not good enough. I wrote some basic documentation, fought with setuptools for packaging, and uploaded it to the Python package server.
Wikked
So lo and behold, here is Wikked, a wiki engine entirely managed with text files sitting in a revision control system.
I think it’s pretty cool, so come read more about it after the break!
Quickstart
You’re too lazy to follow the link to the documentation? Here’s your quick start:
pip install wikked
wk init mywiki
cd mywiki
wk runserver
Text files again?
Yes, this is “Part 2” of my personal crusade to both learn about web technologies and have all my data in text files inside Mercurial or Git. I find it so much easier to manage and backup than some piece of data trapped in an SQL database or something.
It’s obviously not a magic bullet – for one, it doesn’t scale well – but for personal websites I find that it’s perfect.
What’s next
The plan for Wikked is to stabilize it, of course: fix any bugs, make it easier to deploy, make it more configurable. I’m also expecting having to add proper support for Git, as right now only Mercurial is fully supported to store page revisions.
Then, it needs a demo website. There’s one already, actually, but I need to make it a bit more solid, like a cron job that resets it to its original state every night.
Last, I want to get some proper feedback about the Wiki Syntax. It was mostly thrown together as I found I needed something for my own wiki, but I’m still not 100% happy about it.
Fly away, monkeys!
That’s it for now. Be sure to send me some feedback, and to report bugs. Especially the part about reporting bugs, because this thing has never seen any other computer than my laptop and my VPS, so it’s pretty much the mother of all “works on my machine”.
