This blog has been quiet for the past 4 months. And not only that, but most of my open-source projects have been quiet too. I just didn’t feel like working on it.

This is something that Adam Stacoviak and Jerod Santo talk about occasionally on their Changelog podcast¹, whose episodes generally involve interviewing people from the open-source community. I think they called it “project maintainer fatigue”, where an open-source project maintainer basically gets tired, at some point, with the work involved with managing their project.

I have very small projects, and I only dedicate a small amount of my free time to them, so I don’t get this a lot but it does happen every now and then. It usually doesn’t last this long though… I guess I’m getting old.

As is tradition, PieCrust 2.0 was released without much fanfare a few weeks ago. Just like with the previous version, it just happened because, well, nothing happened: I was using PieCrust for a couple other websites without any problem or need for new features, so I figured it might be as good a time as any to make it official.

Time to run pip install -U piecrust.

Since then of course I found a few bugs, so a 2.0.1 release is just around the corner, but that’s just business as usual.

What’s more interesting, first, is that I’ve been a bit quiet over here for the past couple months, and that’s because I was busy with my second hacking project from the holiday break (the first one of which was the recently announced Jouvence). Sadly it didn’t really pan out like I expected, so after working on it for a while I’ve now shelved it for the time being. These things happen.

The second thing is that I started work on PieCrust 3.0. It’s a mess right now but the goal there is to get rid of a bunch of cruft that accumulated over the years. As such, it will break backwards compatibility quite a lot (hence the major version bump) but I’m hopeful the result will be easier to use, easier to
maintain, and maybe even hopefully a bit faster.

I’ve been experimenting on and off with using Vim to write code at work, instead of Visual Studio, and recently I feel like I’ve reached a good enough spot to use it on a daily basis… here it is in action:

Keep reading if you want to know more!

Overview

Here’s the basic setup:

• CtrlP
• my fork of CtrlP Py Matcher
• Gutentags with either
  Exhuberant Ctags or Universal
  Ctags
• p44vim
• YouCompleteMe with clang support
• Some custom plugin (called fb.vim) that ties this all together specifically for Frostbite games codebases.

Vim Plugins

Listing Files

I’m using CtrlP to list files. This is your usual “fast open” thing, where you hit a keyboard shortcut, it opens a list of files, filters it down as you type a “fuzzy” pattern, and then gets out of your way. That’s what you see in the beginning of the video.

CtrlP requires a “root marker” to know what the root of a project is. However, Frostbite games have a lot of crap mixed up with all the source code – configuration scripts, tons of utilities, 3rd party libraries, etc. So asking CtrlP to index all of that would be madness.

The fb.vim plugin therefore generates a “lister script” that lists all the actual source files I care about – engine code, game code, etc. – and I feed that script as CtrlP’s ctrlp_user_command.

Note: Watch out, CtrlP requires passing the project root to user commands, even if you don’t need it because it’s already in some environment variables or something… so always add %s somewhere in ctrlp_user_command, otherwise you’ll get some completely silent errors that will leave you scratching your head for a while.

Pro tip: on Windows, you want to pass /-n /b /s /a-d to dir in order to get some nice output CtrlP can use.

Now you might be asking why I have a plugin generate the script, when I could have written the script directly by hand. Well if I was on a game team, working on the same codebase for 3 years, yeah maybe. But since I’m on the engine team, I have to juggle between dozens of branches, and dozens of different games – so I need the script to be portable, and generating it on Vim startup (when I detect the right environment) is the best way to keep that script up to date with my latest changes.

CtrlP Py-Matcher

Now the problem is performance. Even though I’m only interested in a subset of the codebase, that subset still has around 50k files in it… that’s where CtrlP Py-Matcher comes in.

The default search algorithm for CtrlP is implemented in VimScript so it’s, by definition, as portable as Vim, without the need to install anything. But it’s slow, whereas Py-Matcher is fast, but it’s implemented in Python so you need Python installed… but hey, good news, Python is installed anyways in a Frostbite environment! So yeah, that’s why I’m using that.

There are faster matchers out there for CtrlP (implemented in C/C++) but Windows binaries are hard to find, and Py-Matcher is good enough for this amount of files. Except that it had some Unix-isms that didn’t work well on Windows, so that’s why you may need my fork until my pull request is merged into the main project.

Tags

Ok now I can open files, but I can’t navigate the source code. Typically you need some kind of symbol database, which is where ctags comes in – either Universal or Exhuberant, pick your poison (see links above).

Gutentags (by yours truly) manages the tags file for you, so just install it and be happy. However, it also needs to be told what the root of the project is and, just like with CtrlP, indexing the entirety of the
root folder of a AAA game’s repository would be madness. So the fb.vim plugin (again) generates a “lister script” that will list the same files as for CtrlP, but for Gutentags.

Performance-wise, the resulting tags file contains around a million symbol definitions, and if I need to arbitarily jump to any of those through CtrlP, CtrlP Py-Matcher will take a second but will get there. Not super fast, but interactive enough.

So now I can navigate to any arbitrary tag, from anywhere. Yay!

Perforce

I can open files quickly, and navigate the source. But when I start typing, I need to potentially open that file for editing in Perforce, which is the source control management software that virtually all big game studios use. Unlike cool things like Mercurial or Git, all files in your repository are typically read-only until you explicitely check them out.

Fear not! Here comes P44Vim (by yours truly again), which is a super bare bones plugin for basic Perforce integration. It’s got only a couple commands, and automatically opens read-only file for edit when you start typing.

You don’t really see this part in the video since it happens behind the scenes.

Code Completion

Last but not least, I start typing some code, but there’s no code-completion! Usually I would go: “oh I need to give Intellisense another 10min before it wakes up”… but then I realize I’m not in Visual Studio anymore.

The biggest player in town for this is YouCompleteMe, which uses clang behind the scenes to compile stuff on the fly and give you almost-real-time feedback and completions. I was a bit frightened at first but they really tidied up their install procedure since last time I looked at it many years ago, and their
support for Windows seems pretty good. It only took me a couple minutes to get it going.

…except that, of course, we need to tell it how to compile our code. Once again, that’s where the fb.vim plugin comes in. It will go lurk inside the Visual Studio solutions and extract everything YCM and clang need to know – include directories, preprocessor defines, etc. – about the current file.

All of this happens behind the scenes, but that’s how you see the code completion when I call a method on the data variable, and how, when I later add an incorrect parameter to the function, it gives me feedback right away that this is wrong.

This has some issues, however. Our Windows build is not suposed to be compiled with clang, so I get a lot of warnings, along with a few errors, in the low level code (pragmas, attributes, SSE, etc.). I can pass the right flags to ignore the warnings (a whole bunch of -Wno-foobar), but not the errors. But that’s OK, since clang still builds a mostly correct AST for my current file’s compilation unit, and I still get mostly correct code completion and hints. It’s not perfect, but it’s already better than Intellisense, and faster/lighter than Visual Assist.

Future Work

It’s getting pretty cozy in Vim, here, but there are still a few things to do:

• Setup OmniSharp: I configured YouCompleteMe to give me all the goodness I need in C++, but haven’t looked into it for C# stuff yet.
• Invalid characters in our code: There are quite a lot of files with a UTF BOM header for some reason, and, more puzzling, a few with just invalid UTF8 characters in them. This makes python serialization very upset when Vim passes the tags file to CtrlP Py-Matcher, so I currently have a hacky post-process setup in Gutentags that filters out any symbol with such invalid characters.
• Debugging: I don’t think there’s any getting around Visual Studio for this, and I don’t think anybody would want to anyway since Visual Studio’s debugger is still the best around (don’t get me started on gcc), but I would like to experiment with running an empty Visual Studio (without the whole, super heavy, solution loaded) directly from Vim, with some commands to tell it to open the same file as I have in Vim so I can add breakpoints.

That’s All Folks

If you have feedback, suggestions, or comments about how much of an old bearded nerd I am, please share!

Over the holidays I started 2 hacking projects. This is the first one.

Jouvence is a Python library for parsing and rendering Fountain documents. If you don’t know Fountain, it’s basically Markdown but for writing screenplays. It was created by John August, Nima Yousefi, Stu Maschwitz, and a few other contributors – check out the official website for more information.

The code is, as usual, on both BitBucket and GitHub. The package is on PyPi. The API documentation is hosted on ReadTheDocs.

Here’s how the “Big Fish” screenplay renders with the default HTML renderer:

Of course the whole point of Jouvence is that you write tools and renderers of your own. The Jouvence parser returns a structured document object model that makes it easy to analyze, manipulate, or render screenplays any way you want.

For me, this let me add Fountain support to Wikked, my flat-file wiki engine.

Enjoy!

Happy new year and other such holiday things to the 3 people who read my blog!

I was in San-Francisco for new year’s eve and it was quite lovely.

Apart from the sake of posting something new, this post is also to show off how PieCrust’s admin panel can now upload page assets (in this case, that would be the picture above… check the URL, it’s hosted right here). This feature will be part of the impending 2.0 release, but you can get it by pulling the latest from BitBucket or GitHub as usual.

Cheers!