[Buildbot-commits] [Buildbot] #2480: Thundering herd of pollers overwhelms ssh server, causing nearly silent GitPoller failure

[Buildbot trac]

#2480: Thundering herd of pollers overwhelms ssh server, causing nearly silent
GitPoller failure
----------------------+-----------------------
Reporter:  dank       |      Owner:
    Type:  undecided  |     Status:  new
Priority:  minor      |  Milestone:  undecided
 Version:  0.8.7p1    |   Keywords:
----------------------+-----------------------
 buildbots with more than ten masters using GitPoller against the same git
 server via ssh seem to fail to notice changes.   The only indication of
 error is messages like
   ssh_exchange_identification: Connection closed by remote host
 in twisted.log on the master.

 This happens because the ssh daemon drops incoming connections when they
 exceed the MaxStartups or MaxSessions limits from sshd.conf.  Both default
 to 10.  Users can raise those limits, but it'd be better to avoid the
 thundering herd in the first place.

 A workaround is for the user to fuzz the pollIntervals, e.g.

 GitPoller(repourl,  branches=branchnames, pollinterval=300 +
 random.uniform(-10, 10))

 A fix might be to build the fuzzing into GitPoller or PollingChangeSource,
 so users didn't have to worry about it.
 Perhaps add a pollIntervalFuzz parameter that defaults to 10.

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2480>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation