[Buildbot-devel] differences in test coverage from 0.7.x to 0.8.2?

[Dustin J. Mitchell]

On Thu, Nov 25, 2010 at 11:15 AM, Axel Hecht <l10n.moz at googlemail.com> wrote:
> Is there a track record what made a test flaky?

Well, none of them were "unit" tests - they set up a whole bunch of
Buildbot parts, did some stuff, and then asserted some stuff.  So they
failed quite a bit due to timing problems.  There were lots of
inter-test dependencies, which of course is fun with Twisted because
the tests' ordering can change from run to run.  And dozens of tests
would break on every change because they depended on lots of
implementation details.  I kept a number of the better-behaved tests,
and I've been careful to test new code that is nicely isolated.

> Also, for Firefox, we have had that very discussion a few times, with the
> opposite result, and, I guess, with the opposite practical experience.

Firefox is in a different league, anyway :)

> I consider a good deal of the tests in 0.7.12 to be good examples of
> patterns on how to test for interdependencies between the code one writes
> themselves, and buildbot.

Can you explain that?

Dustin