The Stochastic Game
Ramblings of General Geekery

About unit testing Maya and MStatus macros

posted on

Programmers in the video games and movies industry rarely write unit tests for all kinds of reasons and excuses, but every now and then, it happens. And it can get a bit complicated when you want to test a plug-in hosted by a 3rd party application like Autodesk’s Maya.

Setting up the unit test project

The good thing is, unlike most other 3d modeling packages, Maya comes with built in “batch” and “library” modes. The batch mode effectively runs Maya as a command line program, and the library mode allows you to host most of the Maya core engine inside your own application. This means that, as long as you’re not using anything that depends on the existence of the UI, it’s possible to run automated tests.

Once you’ve created your unit test project using your favourite test framework (lately I’ve been using boost::test), you want to initialize Maya in a minimalist environment (most probably in a test fixture setup phase). You can do that by pointing the MAYA_APP_DIR environment variable to a custom folder of your choice, which would contain a custom Maya.env file, along with maybe a few custom startup scripts. The goal is obviously to not load your complete production environment with all your 3rd party plug-ins. You also probably want to set the MAYA_PLUG_IN_PATH and MAYA_SCRIPT_PATH variables to point to the output or deployment folder(s) of your test project. This can even be done programmatically with the Maya API using MCommonSystemUtils::putEnv, as long as it happens before you initialize the engine.

When you’ve got the correct environment, you can call MLibrary::initialize and wait for an excruciatingly long time (somewhere between 20 and 30 seconds). Ideally, I’d like to start with a fresh Maya engine for each test suite (or even each test!), but given how long it takes for the library to initialize, that’s not an option and I do it only once on start-up. Between tests, I merely clear the scene with a “file -force –new” MEL command. I know you can severely strip down Maya by telling it to not even load the default plug-ins (e.g. if you’re testing something that doesn’t need rendering and animation and skinning and all that, you can in theory not load those features), but I haven’t bothered looking into that yet. If you have, I’d love to hear about it.

Anyway, all this, along with some nice helper functions for checking things in the scene, is pretty much enough to get you going… until you write your first “fail” test, i.e. a test that ensures that a given situation returns the expected error code or throws the expected exception.

The MStatus problem

You see, the problem is that, if you’re a bit paranoid like me, you’re probably using a lot of CHECK_MSTATUS macros around your code to make sure you’ll notice early if anything is not going as expected (the Maya API, being old as it is, doesn’t use exceptions, so if you don’t check return codes regularly you could go a long way with invalid objects, which increases the chances it’ll do some harm to your users’ data). When these macros are passed a non successful MStatus, they print a message to the standard error output. This pretty much means you’ll see that message printed in the terminal window as your unit test runs. That’s fine, but in the case of your fail test you don’t want that to happen because you only want to see unexpected error messages printed in the console.

Looking at the code for those CHECK_MSTATUS macros, one can see that they use the STL’s std::cerr to print the beginning of the message, and then call MStatus::perror to print a user-friendly description of the error. Being naive, I’m thinking “good, I just need to temporarily capture std::cerr!”. So I wrote the little following structure:

struct StreamCapture
{
public:
    StreamCapture(std::ostream& str, const std::ostream& replacement);
    ~StreamCapture();

private:
    std::ostream& mStr;
    const std::ostream& mReplacement;
    std::streambuf* mPrev;
};

StreamCapture::StreamCapture(std::ostream& str, const std::ostream& replacement) :
    mStr(str),
    mReplacement(replacement)
{
    mStr.flush();
    mPrev = mStr.rdbuf();
    mStr.rdbuf(mReplacement.rdbuf());
}

StreamCapture::~StreamCapture()
{
    mStr.flush();
    mStr.rdbuf(mPrev);
}
#define CAPTURE_COUT()
    std::stringstream replace_cout_##__LINE__ ;
    StreamCapture capture_cout_##__LINE__ (std::cout, replace_cout_##__LINE__)

#define CAPTURE_CERR()
    std::stringstream replace_cerr_##__LINE__ ;
    StreamCapture capture_cerr_##__LINE__ (std::cerr, replace_cerr_##__LINE__)

#define CAPTURE_ALL()
    CAPTURE_COUT();
    CAPTURE_CERR()

It’s using the classic RAII pattern, and you can use it as such:

BOOST_AUTO_TEST_CASE(TestThatSomethingFailingDoesntScrewUpTheRest)
{
    // Do something
    initializeStuff();
    {
        CAPTURE_ALL();
        callSomethingThatWillFail();
    }
    // Check that even with the failure, the data in the scene is still ok.
    checkStuff();
}

It won’t take you long to realize that only the first half of any error message printed by CHECK_MSTATUS has been captured. “No worries”, I thought. “They probably print the other half with a good ol’ fprintf, so I’ll capture stderr too!”.

Well… I tried a few different things, from a few different sources, but none of them worked. At best, it didn’t do anything – the error code description would still get printed. At worst, it would seemingly work but would then render the whole program unstable (most of the time it would crash on the second attempt to capture the error output). I don’t know what Maya is using to print that MStatus out, but it sure isn’t a straightforward fprintf.

Now, before moving on to my (somewhat disappointing) solution, let’s look at another MStatus problem.

The CHECK_MSTATUS_AND_RETURN_IT problem

One of the CHECK_MSTATUS macros is CHECK_MSTATUS_AND_RETURN_IT, which checks for any error and, if found, will make the current function return that very same error. A typical use of it would be:

MStatus doSomethingSimple();

MStatus doSomethingMoreComplicated()
{
// Do stuff
CHECK_MSTATUS_AND_RETURN_IT(doSomethingSimple());
// Do more stuff
return MStatus::kSuccess;
}

The problem is that this macro is implemented as such:

#define CHECK_MSTATUS_AND_RETURN_IT(_status)
    CHECK_MSTATUS_AND_RETURN((_status), (_status))

Do you see? It calls CHECK_MSTATUS_AND_RETURN which, if the first status is an error, returns the second status. This means that if the call to doSomethingSimple fails, the macro will call it a second time to get the return value!

This is obviously bad… (and another example of why C++ macros can be evil because it’s not obvious what you can and cannot pass into them).

At first I defined a CHECK_MSTATUS_AND_RETURN_IT_SAFE macro that did the correct thing, but for various reasons I decided to just redefine the original macro and prevent other programmers from making that mistake (at least as long as the my header file was included, which is easier to enforce, especially if you’re using things like pre-compiled headers):

#pragma warning(push)
#pragma warning(disable:4005)
#define CHECK_MSTATUS_AND_RETURN_IT(status)
    CHECK_MSTATUS_AND_RETURN_IT_SAFE(status);
#pragma warning(pop)

Back to the MStatus problem

Now what does this have to do with the error message problem?

Well, now that I was already redefining one of the CHECK_MSTATUS macros, I figured it wasn’t much more trouble to redefine them all (there’s only a handful of them). The only thing you need to do is replace the call to MStatus::perror with a line that sends the MStatus into std::cerr. I mean, MStatus already has operator<< defined for IO streams, so it’s not like I had to do anything more than some copy/paste and changing one line.

So there you have it: my crappy solution for having clean Maya unit tests was to redefine the CHECK_MSTATUS macros so they use the STL’s IO streams. Do you have a better or more elegant solution? I’d love to know about it!