[Buildbot-devel] differences in test coverage from 0.7.x to 0.8.2?

[Axel Hecht]

2010/11/21 Dustin J. Mitchell <dustin at v.igoro.us>

> On Sun, Nov 21, 2010 at 4:08 PM, Axel Hecht <l10n.moz at googlemail.com>
> wrote:
> > My point is, testing coverage from buildbot.master went from 84%
> > testcoverage to 20% (566 tested lines to 155). process.builder from 84%
> (436
> > lines) to 17% (96 lines). Kinda puts me into trouble when John talks
> about
> > how hard it is to update buildbot, as my mozilla genes would talk about
> > tests.
>
> Yes, and the counterargument is that 84% coverage with
> non-operational, flaky tests is worse than 20% coverage with
> operational tests, for the reasons I outlined earlier.  My priority
> for the project is to make sure that others can contribute as
> efficiently and effectively as possible.  The decision to delete most
> of broken_tests was made on that basis, and my experience since
> deleting it has supported the decision.
>

Is there a track record what made a test flaky?

Also, for Firefox, we have had that very discussion a few times, with the
opposite result, and, I guess, with the opposite practical experience.

I know that I myself haven't been the knight in shining armor to fight the
funkyness around at least one test, but still.


> Certainly, having better test coverage in Buildbot would be great, but
> there's a long list of BB would-be-greats.  I'm a relatively slow
> coder, so most remain would-be-greats until someone steps up.  When
> someone does step up, I can guide their work to make sure it has the
> best value possible to the project.  That includes requiring
> documentation and tests for new code, and where possible tests for
> changed code, too.
>
> I'm jealous of your ability to take an idea, bang out a considerable
> amount of code to get a working prototype, and then run from there.
> I'd love to see Buildbot on the receiving end of that code firehose!
>
> To the final point about difficulty in upgrading Buildbot: as Jacob
> Kaplan-Moss pointed out recently
> (http://jacobian.org/writing/buildbot/ci-is-hard/), Buildbot is a
> framework, not an application.  Like many large installs, Mozilla
> pushes that framework particularly hard with a lot of intricate
> master-side logic, so it's natural for upgrading to be risky.  With
> apologies for Mozilla-specific noise on this list, I'll add that (a)
> the 0.8.2 upgrade on Wednesday seems to have gone fairly well and (b)
> there are plans afoot to move much of the logic slave-side, where
> (IMHO) it belongs.
>
>
I consider a good deal of the tests in 0.7.12 to be good examples of
patterns on how to test for interdependencies between the code one writes
themselves, and buildbot. Which is why I'm trying to figure out why they
went where.

Axel


> By the organizational process by which decisions like "let's delete
> most of the tests" are made is a good topic for the December summit.
>
> Dustin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20101125/cf63a347/attachment.html>