[Buildbot-devel] Lock question: Multiple builders, load-balanced slaves

[Roy S. Rapoport]

On Fri, Nov 03, 2006 at 12:18:26AM -0800, Brian Warner wrote:
> That isn't an ideal solution (i.e. I don't think I'd want to put it into the
> mainline code), but I think it might stand a chance of working for this
> problem.

I'm leaving my company next Friday, and I'd like to do everything in my
power to make sure that I leave them with a very robust and well-managed
buildbot environment.  I suspect I'm willing to deploy this code, but only
if you don't think you guys will implement something better in the next few
versions -- I already negotiated with my boss that if we need to wait for
this to be supported in buildbot, I'll come back and put that capability in.

Personally, I think it's useful and makes sense from an OOP perspective to
be able to have the slave tell you "not now, I'm busy."  In fact, It'd make
sense to me to be able to make this a more flexible solution where you
could override some method to do any sort of checking you want.  Imagine if
I was giving a few projects access to my system to do builds, but was
concerned about load -- I'd be able to override this method to get the load
average on the system right now and say "oh, sorry, I'm at load average 1.5
or higher, you can't build here right now."

If that, or something like that, feels like it'd be good, and you think it
could make it into, say, buildbot 0.8, then I'm probably going to be
inclined to wait.  As it is, of our 18 platforms, only one is actually
driving this need right now (most of them take about 3-5 hours to finish
nightly tests; this platform currently takes 9 hours), and we can work
around its slowness by manually load-balancing builds on two systems.   

If you don't think something like this would make it into buildbot in, say,
the next 6-9 months, I'm probably going to have to use the code you've so
kindly suggested.

> We could certainly use some better reporting interfaces to tell when steps or
> builds are waiting on a Lock. (at the moment at work we've got several
> builders sharing a MasterLock (maxCount=2) to limit contention for a slow SVN
> server, and the resulting gaps in the waterfall page are confusing to look
> at). It would probably be sufficient to make the step's status text say
> "waiting for Lock(name)" while it's in that state, but it might also be nice
> to make it look like the step has already started (since by that point we've
> committed to running the step eventually.. it's just a matter of time).

We do that at work on our dashboard -- the way I've got it done is that if
a build is started, but there's no current step, I assume that means we're
waiting for a lock -- and instead of a yellow 'building', I actually show a
blue 'lock-wait.'  My people love it.

-roy