[Buildbot-devel] Lock question: Multiple builders, load-balanced slaves

[Scott Lamb]

On Nov 3, 2006, at 12:18 AM, Brian Warner wrote:
> You want to have a SlaveLock held around
> the whole build (preventing two builders from running  
> simultaneously on a
> given buildslave), but you want the Builders to choose an idle  
> slave rather
> than committing to run on a slave which is running a build (and  
> holding the
> SlaveLock) on behalf of a different Builder.
...
> The SlaveLock (with maxCount=1) would be sufficient to protect the  
> slaves
> against multiple simultaneous builds. But it won't let you obtain full
> utilization of the hardware: sometimes, one of the slaves will be  
> idle when
> there is still work to be done.

Ahh! Put that way, I understand, and I have encountered the same  
problem.

Speaking of which...if you're thinking about tweaking the locks for  
better hardware utilization, there are another couple things I'd be  
thrilled to see along those lines:

(1) BranchLock. Due to the way we manage version numbers, some  
Builders can't be run on the same branch simultaneously. We're  
currently being overly conservative - we use a MasterLock('version'),  
so those builders don't run simultaneously at all, even across branches.

(2) A "step group"...our compile phase is very CPU-intensive, and our  
test phase uses a per-slave resource. We could do something like this:

     cpu = locks.SlaveLock('cpu')
     jtag = locks.SlaveLock('jtag')

     f = factory.BuildFactory([
         s(step.P4, ...),
         s(step.ShellCommand, name='compile', locks=[cpu], ...),
         s(step.ShellCommand, name='test', locks=[jtag], ...)
     ])

...except that we broke the test phase down into three parts, and we  
must not release the lock between. The optimal thing might be like this:

     f = factory.BuildFactory([
         s(step.P4, ...),
         s(step.ShellCommand, name='compile', locks=[cpu], ...),
         StepGroup(name='test', locks=[jtag], steps=[
             s(step.ShellCommand, name='1', ...),
             s(step_twisted.Trial, name='2', ...),
             s(step.ShellCommand, name='3', ...)
         ])
     ])

but since that doesn't exist, we have a single SlaveLock we hold  
during the entire build.

-- 
Scott Lamb <http://www.slamb.org/>