[Buildbot] #2895: Builds take a long time to get started on latent buildslaves

[Buildbot trac]

#2895: Builds take a long time to get started on latent buildslaves
----------------------+-----------------------
Reporter:  agateau    |      Owner:
    Type:  undecided  |     Status:  new
Priority:  major      |  Milestone:  undecided
 Version:  0.8.6p1    |   Keywords:
----------------------+-----------------------
 I have been running Buildbot in a setup with 8 EC2 build slaves, to build
 many (109) projects. The 109 builders are all associated with all slaves.

 When using EC2 latent build slaves, builds take a long time to get
 started. I investigated the code and found out that everytime a latent
 slave is added, {{{maybeStartBuildsForBuilder()}}} is called, causing
 {{{BuildRequestDistributor.maybeStartBuildsOn()}}} to acquire
 {{{pending_builders_lock}}} more than 109 * 8 times. This causes a
 significant delay between the time the buildset is added and the time a
 slave is getting prepared because {{{maybeStartBuildsOn()}}} must release
 all those locks before {{{_activityLoop()}}} can pop a builder.

 On my setup, it can take up to 4 minutes, whereas builds are started
 instantaneously when using classic build slaves.

 Attached patch removes the call to {{{maybeStartBuildsForBuilder()}}} in
 {{{Builder.addLatentSlave()}}} to avoids this delay. The builds are still
 started nevertheless when the slave is ready.

 The patch is against the buildbot-0.8.6 branch as I could not get master
 to run on Debian Wheezy (master requires too many new things), sorry for
 that.

--
Ticket URL: <http://trac.buildbot.net/ticket/2895>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation