[Buildbot-devel] HTML test results

[tom fogal]

Stefan Seefeld <seefeld at sympatico.ca> writes:
> On 03/23/2010 12:18 PM, tom fogal wrote:
> > Hi all, the recent Jinja thread reminded me of a problem I need to
> > figure out for one of my masters.
> >
> > We do a lot of validation based on images and image differencing.
> > Here's an example of something our previous regression testing
> > system generates:
> >
> >    http://portal.nersc.gov/project/visit/10Mar15/davinci_serial/databases_c
> law.html
> >
> > I'd like to create something similar via buildbot.
> >
> > My initial thought is to tar up the directory with those HTML
> > results, then I could FileUpload them to the master, and make them
> > available.  That kind of sucks though, it requires a lot of manual
> > effort for a developer to see what's gone wrong.
>
> I'm not quite sure I understand what you are complaining about, and
> where the perceived effort is.
>
> From your description it sounds like we are doing something
> similar: We let a build run a build, as well as run a test suite. The
> result of that test suite is an html test report. The last step
> in the builder is a DirectoryUpload, so the test report becomes
> available via the build master's web server.  Is that not what you
> are doing ? What part of that requires "a lot of manual effort for a
> developer" ?

I mentioned a tar file, because I forgot about DirectoryUpload.  A tar
file would be a PITA for developers.

> > Perhaps I could upload the images and create a Jinja template which
> > provides the same information as above (.. and would probably look
> > prettier ;)?
>
> In my case, the test report is served by the build master's web
> server.  I can see how this may use some of the jinja magic, though
> it isn't clear to me how that affects the amount of effort in setting
> things up.

As I mention below, the main issue is getting a global view of a test
across a suite of testers.  First of all, each tester is going to name
the image the same, so if I'm going to upload all of them to a single
master, I need to mangle the directory names specially, or at least the
image names.  Secondly, the HTML generated by our test suite is, of
course, ignorant of the results from the other machines.

To construct a global view, I need some kind of foreach loop over all
of the builders associated with a particular schedular.  Each iteration
of the loop needs to pull out the image for *that* builder, which would
again need to know about the name mangling I did during upload.  I
figured that construction could use some jinja, but I really don't know
anything about jinja other than "it's a template engine".  Hence why I
pinged the ML :)

I also need to find a way to do this which is fairly automated, per
test.  We've got something on the order of a hundred tests currently,
which can be run in one of 3 (hopefully 4, soon) modes.  I'm not going
to specify a "DirectoryUpload" for each of them, because that scales
poorly.  Currently I break them up by category; i.e. run all the IO
tests together.  I'm not sure this is best, and so I'm interested in
getting the input of others.

> > One issue is the naming of images.  The failing image is
> > "c_claw_03.png" in the above link; if that failed on two machines,
> > I'd want separate images.  Perhaps a WithProperties could append a
> > machine type ("osx", "osuse-11.1", etc.)?
> >
> > A really excellent start would be a table of, say:
> >
> >    TestCase | Baseline | BuilderName1 | BuilderName2 | BuilderName3 ...
> >
> > Has anyone tackled the issue of distributed/multi-machine
> > image-based testing with buildbot?  How did you do it?
>
> I still believe that buildbot is the wrong tool for managing testing.
> But I know others disagree...

Could you expound?

-tom