[Buildbot-devel] Concurrent builds on different slaves

[Jean-Paul Calderone]

On Tue, 11 Nov 2008 19:39:56 -0500, "Dustin J. Mitchell" <dustin at zmanda.com> wrote:
>On Tue, Nov 11, 2008 at 7:32 PM, Jean-Paul Calderone <exarkun at divmod.com> wrote:
>> Ah.  Also, I've been sharing locks between different slaves like this
>> for a long time and it's never resulted in behavior like I'm now observing.
>> The only thing I haven't done before is use a lock with a builder which has
>> multiple slaves associated with it.
>
>Maybe I don't understand the behavior you're seeing then.  You said
>that it's serializing the builds, which is what I would expect with a
>shared lock.  But I probably jumped to a conclusion :(

I tried to produce a minimal example, but it takes too long to configure
a new buildbot and I gave up after about ten minutes.  So I'll try to
describe the behavior more thoroughly.

There are two slaves, S1 and S2, connected to the master.  There are two
builders configured on the master, B1 and B2.  B1 and B2 both have a
"slavenames" entry which is set to [S1, S2].  B1 and B2 both also have
a "locks" entry which is set to [L1].  L1 is a SlaveLock with maxCount
set to 1.

My expectation is that this will allow B1 to execute on S1 while B2
executes on S2 (or B1/S2 and B2/S1), but prevent B1 and B2 from executing
in parallel on S1 or in parallel on S2.  The documentation says that each
slave has its own copy of each slave lock (or words to that effect), so
I would expect L1 to have an independent acquired/released state on each
of S1 and S2.  I've previously configured masters with such locks, shared
between different builders (none of which had more than one slave, though)
and made observations which are in support of this (actually, the master
I'm talking about has other builders configured in this way and doesn't
have any problem doing those builds in parallel).

What I've actually observed is that:

  - while B1 is running anywhere, sometimes forcing B2 will wait until B1
    has completed, but sometimes not.

  - The assertion error about build numbers which I included in a previous
    message will sometimes appear in the logs.

  - When it does, if B1 was waiting for B2 finish (or vice versa), on the
    waterfall B1 will retain its yellow "Building" header, not actually
    start the build which was waiting, and remain in that state until the
    master is restarted.

  - Sometimes forcing B2 while B1 is running on S1 will cause B2 to be
    scheduled on S1 even though S2 is idle.

I hope that clears up the situation somewhat.  If I can clarify any parts
of the setup any further, please let me know.

Jean-Paul