[Buildbot-devel] py.test and buildbot

[Brian Warner]

> as you can see, our tests seem to pass :o)  But the don't, looking at
> the log, py.test gives us a few Fails.  We'd like the waterfall to be
> red when py.test[1] gives us 1 or more failed tests.
> 
>   It seems that the buildbot 'test' action relies on the return value of
> the command to determine if it should log the test as failing or
> passing, and py.test returns 0 (all normal) unless it dies unexpectedly.

'Test' is a pretty simple step, really intended for systems like the 'make
check' target in a lot of GNU makefiles. For py.test you'll want to write a
slightly more complicated BuildStep that parses the output to determine how
many tests passed/failed. Take a look at step_twisted.Trial for inspiration.

In case it helps, I've got some time scheduled next week to implement py.test
Steps into the buildbot. I'll build upon whatever you've got by that point.
There's a lot more you can do beyond red-vs-green.. it would be nice to get a
list of specific tests that passed or failed, split log output up so you can
display results on a per-test basis, count the number of tests to be run
ahead of time so you can do better ETA prediction, etc.

thanks,
 -Brian