[Buildbot-devel] differences in test coverage from 0.7.x to 0.8.2?

[Dustin J. Mitchell]

On Sun, Nov 21, 2010 at 4:08 PM, Axel Hecht <l10n.moz at googlemail.com> wrote:
> My point is, testing coverage from buildbot.master went from 84%
> testcoverage to 20% (566 tested lines to 155). process.builder from 84% (436
> lines) to 17% (96 lines). Kinda puts me into trouble when John talks about
> how hard it is to update buildbot, as my mozilla genes would talk about
> tests.

Yes, and the counterargument is that 84% coverage with
non-operational, flaky tests is worse than 20% coverage with
operational tests, for the reasons I outlined earlier.  My priority
for the project is to make sure that others can contribute as
efficiently and effectively as possible.  The decision to delete most
of broken_tests was made on that basis, and my experience since
deleting it has supported the decision.

Certainly, having better test coverage in Buildbot would be great, but
there's a long list of BB would-be-greats.  I'm a relatively slow
coder, so most remain would-be-greats until someone steps up.  When
someone does step up, I can guide their work to make sure it has the
best value possible to the project.  That includes requiring
documentation and tests for new code, and where possible tests for
changed code, too.

I'm jealous of your ability to take an idea, bang out a considerable
amount of code to get a working prototype, and then run from there.
I'd love to see Buildbot on the receiving end of that code firehose!

To the final point about difficulty in upgrading Buildbot: as Jacob
Kaplan-Moss pointed out recently
(http://jacobian.org/writing/buildbot/ci-is-hard/), Buildbot is a
framework, not an application.  Like many large installs, Mozilla
pushes that framework particularly hard with a lot of intricate
master-side logic, so it's natural for upgrading to be risky.  With
apologies for Mozilla-specific noise on this list, I'll add that (a)
the 0.8.2 upgrade on Wednesday seems to have gone fairly well and (b)
there are plans afoot to move much of the logic slave-side, where
(IMHO) it belongs.

By the organizational process by which decisions like "let's delete
most of the tests" are made is a good topic for the December summit.

Dustin