[users at bb.net] Evaluating Buildbot for Parallel Building

[Ed Singleton]

I'm evaluating Buildbot as a tool for running our builds, and it looks good
so far, but I just wanted to check that our scenario is a good one for it,
and maybe get a head start on which patterns I should be looking at.

We have a few parts, that I am expecting to always be run manually by
kicking off a build through the web interface, rather than being triggered
from code checkins.

1) Check out several code repos, as well as obtain some data from a remote
service.

2) Perform some standard build steps as shell commands.

3) Run around 30-50 parallel builds, each very similar but with slightly
different parameters.

4) When all the builds have finished (and the number can very each run),
collect all the results and compile them together.

5) Perform more standard build steps as shell commands.


I can see how to do the first half of part 1 and I imagine that there's a
way to pretend that a remote data source is actually a VCS, as that seems
to have been well abstracted.

Parts 2 and 5 I'm pretty sure I've got my head around.  They are just
Builders that create Builds with Steps.

Part 3 I suspect is reasonably easy to do by having one Builder that when
it is triggered, dynamically creates 30-50 Builds, which can then be load
balanced across (say) 10 slaves.  (I'd be very grateful for any tips on
this).

Part 4 I'm not at all sure about.  I can see that each of the 30-50 Builds
can use a Trigger, but I'm not sure how to get Part 4 to wait until it has
seen all the triggers, before it runs.


Does this sound like a good fit for Buildbot?  Or should I be using a
different tool?

Do you know of any open source configurations of Buildbot that do something
similar, so that I can read their code?

Are there any patterns I should be looking at for implementing this?

Thanks in advance

Ed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20160311/833cc175/attachment.html>