[Buildbot-devel] Lock question: Multiple builders, load-balanced slaves

[Roy S. Rapoport]

Imagine this situation:
1. There are two slaves for a given os/hardware;
2. There are four builders that want to use these two slaves; 
3. Each slave can only have one build happening at any given point

I'm trying to figure out how to use interlocks to make this the case.  This
wasn't a problem for me until now, because I wasn't load-balancing across
slaves -- so I just created a lock for each slave and had each builder who
wanted to use that slave try to acquire the lock.

Now, the problem is that even though the lock is a SlaveLock, its actual
usage is in the builder.  So if we have machines MA,MB and builders
BA,BB,BC,BD, assuming interlock LA,LB, if all builders want to acquire both
LA and LB, that means I'm serializing all the builders.  That's wrong,
though -- theoretically, I should be able to have two builds happening in
parallel, one on each system.  Even using buildbot 0.7.5's lock
implementation where you can specify how many builders can acquire a given
lock wouldn't help me, because that doesn't seem to tell buildbot "you can
build on both these systems at the same time, but not more than one builder
per system."

Suggestions? I'm trying to solve the problem where one of our platforms
ends its nightlies at 08:50, and we're thinking load-balancing will help us
do this in a robust way.  

-roy