[Buildbot-devel] Node.js-based testing

[Dustin J. Mitchell]

As Naveen begins implementing the Buildbot frontend in JavaScript, I
got to thinking about how we'd test this new code.  I'd like:
 * tests devs can run in a console, like the existing Python tests
 * good test isolation for unit tests
 * fast - the current tests take 30s or so, and that's already too long

I expect that we'll eventually end up doing some browser automation
with Selenium, to do functional tests.  But those can never meet any
of the criteria above, so let's set them aside for the moment.

I think it makes sense to run the unit tests in nodejs, so I had a
look at some JS testing frameworks.  I had a look at
 - Jasmine
 - Expresso/Mocha
 - Vows
 - Tobi
 - jsdom
 - should.js

Many of them are behavior-driven test frameworks, which I find
ridiculously verbose and really only suited to checking simple things
like 2+2=4.  So I tended to shy away from such frameworks.

I very much liked Mocha's framework, and support for a number of input
styles (I prefer TDD) and output formats.  It has good support for
running all tests under a particular directory.  It *can* run in a
browser, too -- so we could run the unit tests from Selenium later,
alongside the functional tests (with the benefit of avoiding relying
on V8-specific behaviors).

I think should.js has a nice, expressive syntax
(somevariable.should.equal(13)) without reaching BDD's levels of
literary verbosity.  It's a bit more fleixble than assert.

The Buildbot JS will be highly interactive with the server, so we'll
need a fake server.  Tobi seems to provide that -- or, at least, a
browser-like interface to one.  It requires a server, either in node
or at a given hostname and port.  We could either build a fake data
API in node, or run the JS tests against a working Buildbot master,
containing predefiend "fake" data.  Tobi has the advantage of shipping
with jsdom, which can parse HTML and simulate the DOM.

So, presently I'm leaning toward a testing framework composed of
mocha, should.js, Tobi, and jsdom.  This is, of course, with only the
barest glimpse at the new JS code from Naveen.

I'm new to this area, so any suggestions or comments (or code!) from
those more knowledgeable than me will be *highly* appreciated.

Dustin