[Buildbot-devel] Lock question: Multiple builders, load-balanced slaves

Scott Lamb slamb at slamb.org
Fri Nov 3 21:24:03 UTC 2006

On Nov 3, 2006, at 12:18 AM, Brian Warner wrote:
> You want to have a SlaveLock held around
> the whole build (preventing two builders from running  
> simultaneously on a
> given buildslave), but you want the Builders to choose an idle  
> slave rather
> than committing to run on a slave which is running a build (and  
> holding the
> SlaveLock) on behalf of a different Builder.
> The SlaveLock (with maxCount=1) would be sufficient to protect the  
> slaves
> against multiple simultaneous builds. But it won't let you obtain full
> utilization of the hardware: sometimes, one of the slaves will be  
> idle when
> there is still work to be done.

Ahh! Put that way, I understand, and I have encountered the same  

Speaking of which...if you're thinking about tweaking the locks for  
better hardware utilization, there are another couple things I'd be  
thrilled to see along those lines:

(1) BranchLock. Due to the way we manage version numbers, some  
Builders can't be run on the same branch simultaneously. We're  
currently being overly conservative - we use a MasterLock('version'),  
so those builders don't run simultaneously at all, even across branches.

(2) A "step group"...our compile phase is very CPU-intensive, and our  
test phase uses a per-slave resource. We could do something like this:

     cpu = locks.SlaveLock('cpu')
     jtag = locks.SlaveLock('jtag')

     f = factory.BuildFactory([
         s(step.P4, ...),
         s(step.ShellCommand, name='compile', locks=[cpu], ...),
         s(step.ShellCommand, name='test', locks=[jtag], ...)

...except that we broke the test phase down into three parts, and we  
must not release the lock between. The optimal thing might be like this:

     f = factory.BuildFactory([
         s(step.P4, ...),
         s(step.ShellCommand, name='compile', locks=[cpu], ...),
         StepGroup(name='test', locks=[jtag], steps=[
             s(step.ShellCommand, name='1', ...),
             s(step_twisted.Trial, name='2', ...),
             s(step.ShellCommand, name='3', ...)

but since that doesn't exist, we have a single SlaveLock we hold  
during the entire build.

Scott Lamb <http://www.slamb.org/>

More information about the devel mailing list