[Buildbot-devel] HTML test results

[Stefan Seefeld]

On 03/23/2010 12:18 PM, tom fogal wrote:
> Hi all, the recent Jinja thread reminded me of a problem I need to
> figure out for one of my masters.
>
> We do a lot of validation based on images and image differencing.
> Here's an example of something our previous regression testing system
> generates:
>
>    http://portal.nersc.gov/project/visit/10Mar15/davinci_serial/databases_claw.html
>
> I'd like to create something similar via buildbot.
>
> My initial thought is to tar up the directory with those HTML results,
> then I could FileUpload them to the master, and make them available.
> That kind of sucks though, it requires a lot of manual effort for a
> developer to see what's gone wrong.
>    

I'm not quite sure I understand what you are complaining about, and 
where the perceived effort is.

 From your description it sounds like we are doing something similar: We 
let a build run a build, as well as run a test suite. The result of that 
test suite is an html test report. The last step in the builder is a 
DirectoryUpload, so the test report becomes available via the build 
master's web server.
Is that not what you are doing ? What part of that requires "a lot of 
manual effort for a developer" ?


> Perhaps I could upload the images and create a Jinja template which
> provides the same information as above (.. and would probably look
> prettier ;)?

In my case, the test report is served by the build master's web server. 
I can see how this may use some of the jinja magic, though it isn't 
clear to me how that affects the amount of effort in setting things up.


>   One issue is the naming of images.  The failing image is
> "c_claw_03.png" in the above link; if that failed on two machines, I'd
> want separate images.  Perhaps a WithProperties could append a machine
> type ("osx", "osuse-11.1", etc.)?
>
> A really excellent start would be a table of, say:
>
>    TestCase | Baseline | BuilderName1 | BuilderName2 | BuilderName3 ...
>
> Has anyone tackled the issue of distributed/multi-machine image-based
> testing with buildbot?  How did you do it?
>    

I still believe that buildbot is the wrong tool for managing testing. 
But I know others disagree...

     Stefan

-- 

       ...ich hab' noch einen Koffer in Berlin...