[Buildbot-devel] RFE: load management for build slaves

[Stefan Seefeld]

John Pye wrote:
> Hi Stefan
> 
> Your problem sounds a lot like the one tackled by queuing systems in
> beowulf clusters. I think that there is a some reasonably mature
> software for this, so maybe there are be some wheels that could be
> reused in this.

Certainly.

> I have to say that I think that the paradigm for
> BuildBot is rather different that that of a queuing system. In BuildBot,
> the assumption is that you have a whoel stack of different machines with
> different compilers/architecture, etc, so when you specify a task,
> you're pretty clear that you want that task to be run on the specified
> machine.

That is true when you have only one machine per 'host type' (as discussed
in my original mail). I don't think buildbot can be compared to a queuing
system. Both have different goals. For buildbot, a queuing system would be
the means, not the end.

In fact, it might be possible to plug in a queuing system into a buildbot
setup such that a particular step submits a task to a queuing system and
reports back completion when it is notified by the queuing system.

However, I'm not talking about big homogenous clusters, but rather, of
setups where a small number of machines is configured equally and thus
could share the load.
And even if there is just a single machine available, it is important
to be able to limit the number of concurrently executed builders, to
make the machine responsive to other tasks it may be assigned to.

Regards,
		Stefan