[Buildbot-commits] [Buildbot] #1702: The new slave disconnect timeout in BotMaster.getPerspective is too optimistic

[Buildbot]

#1702: The new slave disconnect timeout in BotMaster.getPerspective is too
optimistic
------------------------+---------------------------------------------------
Reporter:  exarkun      |       Owner:           
    Type:  enhancement  |      Status:  new      
Priority:  major        |   Milestone:  undecided
 Version:  0.8.2        |    Keywords:           
------------------------+---------------------------------------------------
 When handling a new connection to a slave which already has a connection,
 the new connection is always rejected.  I think the idea here is that the
 slave will retry shortly, and if the old connection has since died, then
 the new connection will get to own the slave.

 The new connection is allowed to live for 5 seconds though (

             def kill():
                 log.msg("killing new slave on", new_tport.getPeer())
                 new_tport.loseConnection()
             reactor.callLater(5, kill)

 ), I believe in an attempt to reduce log spam and unnecessary work, as it
 will take some time before the old connection dies in certain
 circumstances (if the network hiccuped, it could be 10 or 20 minutes).  A
 five second delay means the slave may try as many as 240 times before the
 old connection is gone.

 Increasing this delay to 60 or 120 seconds would be nice.  Even nicer,
 though, would be to hold on to the new slave connection until the ping
 succeeds or fails.  If it fails, the new slave connection can be accepted
 immediately, without forcing another connection attempt.  If it succeeds,
 then an even longer delay might be imposed, or something entirely
 different could be tried: hold the connection in reserve in case the old
 one dies, then switch to it immediately; hold it open, but don't ever use
 it, instead email its administrator and tell them something weird is going
 on; send it a graceful shutdown command.

-- 
Ticket URL: <http://buildbot.net/trac/ticket/1702>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation