[Buildbot-devel] differences in test coverage from 0.7.x to 0.8.2?

[Marc-Antoine Ruel]

Le 25 novembre 2010 11:15, Axel Hecht <l10n.moz at googlemail.com> a écrit :

> 2010/11/21 Dustin J. Mitchell <dustin at v.igoro.us>
>
>> On Sun, Nov 21, 2010 at 4:08 PM, Axel Hecht <l10n.moz at googlemail.com>
>> wrote:
>>
>> > My point is, testing coverage from buildbot.master went from 84%
>> > testcoverage to 20% (566 tested lines to 155). process.builder from 84%
>> (436
>> > lines) to 17% (96 lines). Kinda puts me into trouble when John talks
>> about
>> > how hard it is to update buildbot, as my mozilla genes would talk about
>> > tests.
>>
>> Yes, and the counterargument is that 84% coverage with
>> non-operational, flaky tests is worse than 20% coverage with
>> operational tests, for the reasons I outlined earlier.  My priority
>> for the project is to make sure that others can contribute as
>> efficiently and effectively as possible.  The decision to delete most
>> of broken_tests was made on that basis, and my experience since
>> deleting it has supported the decision.
>>
>
> Is there a track record what made a test flaky?
>
> Also, for Firefox, we have had that very discussion a few times, with the
> opposite result, and, I guess, with the opposite practical experience.
>

Interesting, for Chromium, we kill flaky tests. In fact we mark them as
flaky with a tag and eventually kill them if they aren't fixed.

The main issue we faced w.r.t. flaky tests is that the flakiness is induced
by another part of the code that is not strictly related to this test and in
general it's hard to diagnose by definition.



> I know that I myself haven't been the knight in shining armor to fight the
> funkyness around at least one test, but still.
>
>
>> Certainly, having better test coverage in Buildbot would be great, but
>> there's a long list of BB would-be-greats.  I'm a relatively slow
>> coder, so most remain would-be-greats until someone steps up.  When
>> someone does step up, I can guide their work to make sure it has the
>> best value possible to the project.  That includes requiring
>> documentation and tests for new code, and where possible tests for
>> changed code, too.
>>
>> I'm jealous of your ability to take an idea, bang out a considerable
>> amount of code to get a working prototype, and then run from there.
>> I'd love to see Buildbot on the receiving end of that code firehose!
>>
>> To the final point about difficulty in upgrading Buildbot: as Jacob
>> Kaplan-Moss pointed out recently
>> (http://jacobian.org/writing/buildbot/ci-is-hard/), Buildbot is a
>> framework, not an application.  Like many large installs, Mozilla
>> pushes that framework particularly hard with a lot of intricate
>> master-side logic, so it's natural for upgrading to be risky.  With
>> apologies for Mozilla-specific noise on this list, I'll add that (a)
>> the 0.8.2 upgrade on Wednesday seems to have gone fairly well and (b)
>> there are plans afoot to move much of the logic slave-side, where
>> (IMHO) it belongs.
>>
>>
> I consider a good deal of the tests in 0.7.12 to be good examples of
> patterns on how to test for interdependencies between the code one writes
> themselves, and buildbot. Which is why I'm trying to figure out why they
> went where.
>

I tend to agree that end-to-end smoke tests are still valuable. But I don't
think my opinion values much here.

M-A

Axel
>
>
>> By the organizational process by which decisions like "let's delete
>> most of the tests" are made is a good topic for the December summit.
>>
>> Dustin
>>
>
>
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
> Tap into the largest installed PC base & get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20101125/f1f014e9/attachment.html>