[Buildbot-devel] HTML test results

[tom fogal]

Hi all, the recent Jinja thread reminded me of a problem I need to
figure out for one of my masters.

We do a lot of validation based on images and image differencing.
Here's an example of something our previous regression testing system
generates:

  http://portal.nersc.gov/project/visit/10Mar15/davinci_serial/databases_claw.html

I'd like to create something similar via buildbot.

My initial thought is to tar up the directory with those HTML results,
then I could FileUpload them to the master, and make them available.
That kind of sucks though, it requires a lot of manual effort for a
developer to see what's gone wrong.

Perhaps I could upload the images and create a Jinja template which
provides the same information as above (.. and would probably look
prettier ;)? One issue is the naming of images.  The failing image is
"c_claw_03.png" in the above link; if that failed on two machines, I'd
want separate images.  Perhaps a WithProperties could append a machine
type ("osx", "osuse-11.1", etc.)?

A really excellent start would be a table of, say:

  TestCase | Baseline | BuilderName1 | BuilderName2 | BuilderName3 ...

Has anyone tackled the issue of distributed/multi-machine image-based
testing with buildbot?  How did you do it?

-tom