[Buildbot-devel] Ensuring test success rate increases on legacy code

[Neil Hemingway]

Hi,

	we have an old system, for which many of the tests work in some
branches and fail in others.  Obviously we want all tests to work for
all branches, but until we get there, I would like to have buildbot
notify us if:
* A branch was passing all the tests, but now doesn't
* A branch was failing some tests, but now passes all of them
* A branch is now failing more tests than it did before.

I've already done some code diving but before I go too far, is there an
easy way that I've missed of having bb do this kind of thing?

If not, then I propose doing the following.  Can anyone see any holes in
it?
* Subclass ShellCommand (or Test), creating a new createSummary that
peruses log.getText to parse out the failure count.
* Retrieve the previous "gold standard" failure count for this branch
from somewhere
* Set some variables on the builder (previous and new failure count)
that IrcStatusBot can get at to report on
* Compare.  If fewer tests failed, then store as the new gold standard.

* Subclass IrcStatusBot to report if any of the following are true:
  + new failure count > previous failure count
  + new failure count == 0 and previous failure count > 0
  + new failure count > 0 and previous failure count == 0

Regards,
Neil