The Stochastic Game

Ramblings of General Geekery

My door opening on a cheap ghost that I move around on a string is surprisingly effective on kids so far šŸ˜…


Lately I’ve been working to make PieCrust work with the Micropub protocol… so, let’s just say if this shows up: it works! šŸ™‚


I didn’t feel like working on it

This blog has been quiet for the past 4 months. And not only that, but most of my open-source projects have been quiet too. I just didn’t feel like working on it.

This is something that Adam Stacoviak and Jerod Santo talk about occasionally on their Changelog podcast1, whose episodes generally involve interviewing people from the open-source community. I think they called it “project maintainer fatigue”, where an open-source project maintainer basically gets tired, at some point, with the work involved with managing their project.

I have very small projects, and I only dedicate a small amount of my free time to them, so I don’t get this a lot but it does happen every now and then. It usually doesn’t last this long though… I guess I’m getting old.


  1. Oh yeah during the hiatus I started listening to podcasts, among other things. ↩︎


So… I’m going to start microblogging here since Micro.blog (which I backed on Kickstarter) has finally launched. Let’s see how that goes.


Piecrust 2.0 and Beyond

As is tradition, PieCrust 2.0 was released without much fanfare a few weeks ago. Just like with the previous version, it just happened because, well, nothing happened: I was using PieCrust for a couple other websites without any problem or need for new features, so I figured it might be as good a time as any to make it official.

Time to run pip install -U piecrust.

Since then of course I found a few bugs, so a 2.0.1 release is just around the corner, but thatā€™s just business as usual.

Whatā€™s more interesting, first, is that Iā€™ve been a bit quiet over here for the past couple months, and thatā€™s because I was busy with my second hacking project from the holiday break (the first one of which was the recently announced Jouvence). Sadly it didnā€™t really pan out like I expected, so after working on it for a while Iā€™ve now shelved it for the time being. These things happen.

The second thing is that I started work on PieCrust 3.0. Itā€™s a mess right now but the goal there is to get rid of a bunch of cruft that accumulated over the years. As such, it will break backwards compatibility quite a lot (hence the major version bump) but Iā€™m hopeful the result will be easier to use, easier to
maintain, and maybe even hopefully a bit faster.


AAA Gamedev with Vim

Iā€™ve been experimenting on and off with using Vim to write code at work, instead of Visual Studio, and recently I feel like Iā€™ve reached a good enough spot to use it on a daily basisā€¦ here it is in action:

Keep reading if you want to know more!

Overview

Hereā€™s the basic setup:

Vim Plugins

Listing Files

Iā€™m using CtrlP to list files. This is your usual ā€œfast openā€ thing, where you hit a keyboard shortcut, it opens a list of files, filters it down as you type a ā€œfuzzyā€ pattern, and then gets out of your way. Thatā€™s what you see in the beginning of the video.

CtrlP requires a ā€œroot markerā€ to know what the root of a project is. However, Frostbite games have a lot of crap mixed up with all the source code ā€“ configuration scripts, tons of utilities, 3rd party libraries, etc. So asking CtrlP to index all of that would be madness.

The fb.vim plugin therefore generates a ā€œlister scriptā€ that lists all the actual source files I care about ā€“ engine code, game code, etc. ā€“ and I feed that script as CtrlPā€™s ctrlp_user_command.

Note: Watch out, CtrlP requires passing the project root to user commands, even if you donā€™t need it because itā€™s already in some environment variables or somethingā€¦ so always add %s somewhere in ctrlp_user_command, otherwise youā€™ll get some completely silent errors that will leave you scratching your head for a while.

Pro tip: on Windows, you want to pass /-n /b /s /a-d to dir in order to get some nice output CtrlP can use.

Now you might be asking why I have a plugin generate the script, when I could have written the script directly by hand. Well if I was on a game team, working on the same codebase for 3 years, yeah maybe. But since Iā€™m on the engine team, I have to juggle between dozens of branches, and dozens of different games ā€“ so I need the script to be portable, and generating it on Vim startup (when I detect the right environment) is the best way to keep that script up to date with my latest changes.

CtrlP Py-Matcher

Now the problem is performance. Even though Iā€™m only interested in a subset of the codebase, that subset still has around 50k files in itā€¦ thatā€™s where CtrlP Py-Matcher comes in.

The default search algorithm for CtrlP is implemented in VimScript so itā€™s, by definition, as portable as Vim, without the need to install anything. But itā€™s slow, whereas Py-Matcher is fast, but itā€™s implemented in Python so you need Python installedā€¦ but hey, good news, Python is installed anyways in a Frostbite environment! So yeah, thatā€™s why Iā€™m using that.

There are faster matchers out there for CtrlP (implemented in C/C++) but Windows binaries are hard to find, and Py-Matcher is good enough for this amount of files. Except that it had some Unix-isms that didnā€™t work well on Windows, so thatā€™s why you may need my fork until my pull request is merged into the main project.

Tags

Ok now I can open files, but I canā€™t navigate the source code. Typically you need some kind of symbol database, which is where ctags comes in ā€“ either Universal or Exhuberant, pick your poison (see links above).

Gutentags (by yours truly) manages the tags file for you, so just install it and be happy. However, it also needs to be told what the root of the project is and, just like with CtrlP, indexing the entirety of the
root folder of a AAA gameā€™s repository would be madness. So the fb.vim plugin (again) generates a ā€œlister scriptā€ that will list the same files as for CtrlP, but for Gutentags.

Performance-wise, the resulting tags file contains around a million symbol definitions, and if I need to arbitarily jump to any of those through CtrlP, CtrlP Py-Matcher will take a second but will get there. Not super fast, but interactive enough.

So now I can navigate to any arbitrary tag, from anywhere. Yay!

Perforce

I can open files quickly, and navigate the source. But when I start typing, I need to potentially open that file for editing in Perforce, which is the source control management software that virtually all big game studios use. Unlike cool things like Mercurial or Git, all files in your repository are typically read-only until you explicitely check them out.

Fear not! Here comes P44Vim (by yours truly again), which is a super bare bones plugin for basic Perforce integration. Itā€™s got only a couple commands, and automatically opens read-only file for edit when you start typing.

You donā€™t really see this part in the video since it happens behind the scenes.

Code Completion

Last but not least, I start typing some code, but thereā€™s no code-completion! Usually I would go: ā€œoh I need to give Intellisense another 10min before it wakes upā€ā€¦ but then I realize Iā€™m not in Visual Studio anymore.

The biggest player in town for this is YouCompleteMe, which uses clang behind the scenes to compile stuff on the fly and give you almost-real-time feedback and completions. I was a bit frightened at first but they really tidied up their install procedure since last time I looked at it many years ago, and their
support for Windows seems pretty good. It only took me a couple minutes to get it going.

ā€¦except that, of course, we need to tell it how to compile our code. Once again, thatā€™s where the fb.vim plugin comes in. It will go lurk inside the Visual Studio solutions and extract everything YCM and clang need to know ā€“ include directories, preprocessor defines, etc. ā€“ about the current file.

All of this happens behind the scenes, but thatā€™s how you see the code completion when I call a method on the data variable, and how, when I later add an incorrect parameter to the function, it gives me feedback right away that this is wrong.

This has some issues, however. Our Windows build is not suposed to be compiled with clang, so I get a lot of warnings, along with a few errors, in the low level code (pragmas, attributes, SSE, etc.). I can pass the right flags to ignore the warnings (a whole bunch of -Wno-foobar), but not the errors. But thatā€™s OK, since clang still builds a mostly correct AST for my current fileā€™s compilation unit, and I still get mostly correct code completion and hints. Itā€™s not perfect, but itā€™s already better than Intellisense, and faster/lighter than Visual Assist.

Future Work

Itā€™s getting pretty cozy in Vim, here, but there are still a few things to do:

  • Setup OmniSharp: I configured YouCompleteMe to give me all the goodness I need in C++, but havenā€™t looked into it for C# stuff yet.
  • Invalid characters in our code: There are quite a lot of files with a UTF BOM header for some reason, and, more puzzling, a few with just invalid UTF8 characters in them. This makes python serialization very upset when Vim passes the tags file to CtrlP Py-Matcher, so I currently have a hacky post-process setup in Gutentags that filters out any symbol with such invalid characters.
  • Debugging: I donā€™t think thereā€™s any getting around Visual Studio for this, and I donā€™t think anybody would want to anyway since Visual Studioā€™s debugger is still the best around (donā€™t get me started on gcc), but I would like to experiment with running an empty Visual Studio (without the whole, super heavy, solution loaded) directly from Vim, with some commands to tell it to open the same file as I have in Vim so I can add breakpoints.

Thatā€™s All Folks

If you have feedback, suggestions, or comments about how much of an old bearded nerd I am, please share!


Announcing Jouvence

Over the holidays I started 2 hacking projects. This is the first one.

Jouvence is a Python library for parsing and rendering Fountain documents. If you donā€™t know Fountain, itā€™s basically Markdown but for writing screenplays. It was created by John August, Nima Yousefi, Stu Maschwitz, and a few other contributors ā€“ check out the official website for more information.

The code is, as usual, on both BitBucket and GitHub. The package is on PyPi. The API documentation is hosted on ReadTheDocs.

Hereā€™s how the ā€œBig Fishā€ screenplay renders with the default HTML renderer:

Of course the whole point of Jouvence is that you write tools and renderers of your own. The Jouvence parser returns a structured document object model that makes it easy to analyze, manipulate, or render screenplays any way you want.

For me, this let me add Fountain support to Wikked, my flat-file wiki engine.

Enjoy!


Happy New Year!

Happy new year and other such holiday things to the 3 people who read my blog!

I was in San-Francisco for new yearā€™s eve and it was quite lovely.

Apart from the sake of posting something new, this post is also to show off how PieCrustā€™s admin panel can now upload page assets (in this case, that would be the picture aboveā€¦ check the URL, itā€™s hosted right here). This feature will be part of the impending 2.0 release, but you can get it by pulling the latest from BitBucket or GitHub as usual.

Cheers!