[Buildbot-devel] differences in test coverage from 0.7.x to 0.8.2?

[Axel Hecht]

2010/11/25 Dustin J. Mitchell <dustin at v.igoro.us>

> On Thu, Nov 25, 2010 at 11:15 AM, Axel Hecht <l10n.moz at googlemail.com>
> wrote:
> > Is there a track record what made a test flaky?
>
> Well, none of them were "unit" tests - they set up a whole bunch of
> Buildbot parts, did some stuff, and then asserted some stuff.  So they
> failed quite a bit due to timing problems.  There were lots of
> inter-test dependencies, which of course is fun with Twisted because
> the tests' ordering can change from run to run.  And dozens of tests
> would break on every change because they depended on lots of
> implementation details.  I kept a number of the better-behaved tests,
> and I've been careful to test new code that is nicely isolated.
>

That's somewhat clearer,  yet I miss the action to regain ground on test
coverage. As it is, I wouldn't want to try to write patches on the 0.8.x
code base. Also, I think that there are a good deal of tests that slipped
between the cracks.

> Also, for Firefox, we have had that very discussion a few times, with the
> > opposite result, and, I guess, with the opposite practical experience.
>
> Firefox is in a different league, anyway :)
>

I don't think so, at least not in terms of tests. I think there are some
parallels between "pure unit" and xpcshell tests, many of the 0.7.12 tests
and mochitest, and maybe there's even a parallel for mozmill. To rephrase
that totally for those on the thread that are not deeply into the Firefox
testing lingo, both Firefox and buildbot are network-aware, event-driven,
single-thread-event-loop, callback-based systems. The coding patterns that
are hard to test or easy to test are rather similar, AFAICT.

dbaron wrote a pretty pointed message regarding what a test should achieve,
in his opinion, on
http://groups.google.com/group/mozilla.dev.tree-management/msg/02a3573ac6dfa99f.
And guess, I agree :-)



> I consider a good deal of the tests in 0.7.12 to be good examples of
> patterns on how to test for interdependencies between the code one writes
> themselves, and buildbot.

 Can you explain that?
>

I think that testing overall functionality is key. In particular if we're
considering buildbot to be a CI platform, there are promises made that
should be verified. From as simple things as "it runs two builds" to "it
runs builds in parallel" to "if my change has a property, and my build
request has a property, the build ends up with ...".

Users of the platform can then take tests for functionality they depend on,
integrate them into their own test suite, and on an update of the platform,
run their test suite against the new platform and be at least highly
comfortable that their code is going to work. Or, if the tests fail, they
can figure out what changes were made to the upstream tests, and how that
affects their own intergration with the platform.

Axel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20101129/44f300ae/attachment.html>