[Buildbot-devel] differences in test coverage from 0.7.x to 0.8.2?

[Dustin J. Mitchell]

On Fri, Nov 19, 2010 at 8:03 PM, Axel Hecht <l10n.moz at googlemail.com> wrote:
> I've been having day-dreams about stuff on top of buildbot, mostly based on
> tests I knew existed on 0.7.x.
>
> Now, I've tried to find tests like those on 0.8.2, and I'm not successful.
> Like, no tests mentioning dummy.Dummy or Triggerable.
>
> Did those tests go anywhere? And if so, is there a reason they did?

Yes, about a year ago we blew away all of the flaky, unreliable tests,
because they were causing more harm than good:
 * downstream packagers were delaying releases due to failing tests,
and didn't appreciate the "oh, yeah, those tests are flaky" excuse
 * contributors were confused that tests were flaking when they added new code
 * new contributions would break unrelated tests that made invalid assumptions
 * the tests were not at all isolated, so "fixing" = "completely rewriting"

This was a controversial decision, but I think that history has shown
it to be a net positive.  As new, better-isolated code is written,
it's getting new tests.  We've also built some better frameworks to
allow more natural isolation of units for testing.

I'm not surprised there aren't tests for dummy.Dummy, since that's not
in the codebase :)

Dustin