[Buildbot-devel] Determining steps based on what's checked out...

[Greg Ward]

On 16 March 2012, Jarod McBride said:
> So let's say I have a project that has a list of components in it. 
> There's a folder for each component and a set of unit tests.  Each
> component's test can be run by calling...
> 
> phantomjs foo/build/scripts/jasmineRunner.js http://localhost:4020/"component name"/current/en/tests/unit.html
> 
> So I'd like to have a step that looks at how many components there
> are in the git checkout, then creates a buildstep for each component
> and runs that unit test based on the name of the component folder. 
> Like I said, I can pawn the work off to a shell script to do it but
> then I don't get the same level of reporting or control in buildbot
> over those individual steps.

I had a similar need a couple of years ago, and I hacked something
together by doing crazy stuff with logs.

The requirement was: a checkout of 60 or 70 RPM .spec files. Build 'em
all and report which ones failed. Adapt automatically to added/removed
.spec files (duh). I already had a big Python script that was smart
enough to loop over all the .spec files and build each one with a
separate invocation of rpmbuild. This is obviously pretty analogous to
your directories of unit tests.

So here's what I did:

  * added an option to my loop-over-the-spec-files script to write a
    separate log file for each build: apache.spec generated
    logs/apache.log, python.spec generated logs/python.log, etc.

  * added an option to the script to make it segregate logs from
    failed builds in a different directory

  * modify my BuildBot config to use both options (otherwise, why
    bother adding them?)

  * add build steps to copy the failed log files from slave to master
    -- I ran tar on the slave, then a FileUpload to get it to the
    master, because I was using BuildBot 0.7.9 with no 
    DirectoryUpload step

  * add a custom build step to unpack the tar file of failed logs on
    the master (oh yeah, now I remember why DirectoryUpload would have
    been better if I had had it)

  * on the master, loop over the unpacked failed logs and turn each one
    into a separate log object attached to the "unpack and dump logs"
    build step; the build failed if there were any failed logs. 
    I presume the build would have succeeded if there were no failed
    logs, but alas! we've never actually been able to build all of
    our third-party RPMs at the same time ;-(

End result: one long build step of building everything, followed by a
fast build step of copying-and-unpacking logs, which is where the
failure logs actually were. It wasn't quite ideal, but it got the job
done. And it made it easy to look only at failed builds.

I'll attach the source file that did all this. No guarantees, YMMV, it
worked 3 years ago with BuildBot 0.7.x, etc.

        Greg
-- 
Greg Ward                                http://www.gerg.ca/
I'd like some JUNK FOOD ... and then I want to be ALONE --
-------------- next part --------------
A non-text attachment was scrubbed...
Name: buildrpms.py
Type: text/x-python
Size: 10649 bytes
Desc: not available
URL: <http://buildbot.net/pipermail/devel/attachments/20120319/35a63f71/attachment.py>