[Buildbot-devel] RFE: load management for build slaves

[John Pye]

Hi Stefan

Your problem sounds a lot like the one tackled by queuing systems in
beowulf clusters. I think that there is a some reasonably mature
software for this, so maybe there are be some wheels that could be
reused in this. I have to say that I think that the paradigm for
BuildBot is rather different that that of a queuing system. In BuildBot,
the assumption is that you have a whoel stack of different machines with
different compilers/architecture, etc, so when you specify a task,
you're pretty clear that you want that task to be run on the specified
machine.

On the other hand, in a queuing system, the basic assumption is that all
the machines in your cluster can perform the same task, so it's a case
of allocating the queued tasks to the available machine such that they
all get completed in the shorted possible time, or the most important
ones get completed in the shortest possible time, or whatever, depending
on your queue policy.

So I'm not saying that BuildBot couldn't be adapted for this, and I
really don't know the internals of BuildBot that well other than what I
absorbed from setting up my configuration file, but it does seem to me
that it might have a slightly different concept behind it. I'll be
interested to see how this goes.

BTW here are some links
http://www.ac3.edu.au/edu/PBS/index.html
http://www.openpbs.org/about.html
http://beowulf.cheme.cmu.edu/cgi-bin/unixhelp.cgi#chapter_09

Cheers
JP

Stefan Seefeld wrote:

> hi there,
>
> I'v been using buildbot 0.7.2 quite successfully for a couple of weeks
> now.
> It's a great tool to automate build infrastructure transparently.
>
> As we are looking into moving more of our build infrastructure to
> buildbot,
> I'm a little concerned about scalability. Other then explicitely setting
> build times (via Periodic or Nightly) there doesn't appear to be any way
> to manage buildslave loads.
>
> I'm thus looking for ways to improve this situation, and I can see two
> alternative approaches:
> 1) limitting concurrency on a slave and
> 2) schedule based on current load
>
> Both have advantages and disadvantages.
>
> Limitting concurrency:
>
> A slave would have some 'concurrency' number assigned to it, reflecting
> how many parallel builders are allowed to run on it in parallel. In its
> simplest implementation, a slave would queue builders sent to it to
> enforce that limit. Alternatively, the buildmaster / scheduler could
> do the queuing on the master side.
>
> While this seems reasonably straight forward to implement (at least the
> slave-side queuing), the obvious drawback here is that the logic breaks
> down as soon as there are multiple slaves on the same host.
>
> Load balancing:
>
> A slave provides the means for the master to query a 'load level', based
> on which the master decides whether or not to run a builder there.
> While it would be possible to just queue tasks and then ping slaves
> regularly to watch for better times, I believe the best way to support
> load balancing would be to allow multiple slaves to be classified
> equally,
> so that builders can run on any slave in a given class. So instead of
> assigning builders to slaves, slaves would get some new 'host type' (say)
> attribute, to which builders then get assigned. Then the owning scheduler
> can pick the best host to run the builder on, or queue it if none is
> available.
>
> What do people think about the above ? Would this be a valuable addition
> to buildbot ? What is the best way to implement it ? How could it be
> broken down into subtasks ?
>
> Regards,
>         Stefan
>
>
> -------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache
> Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>