[Buildbot] #3360: "Never saw reconfiguration finish." due to log rotation

[Buildbot trac]

#3360: "Never saw reconfiguration finish." due to log rotation
--------------------+-----------------------
Reporter:  dcoshea  |      Owner:
    Type:  defect   |     Status:  new
Priority:  minor    |  Milestone:  undecided
 Version:  master   |   Keywords:
--------------------+-----------------------
 I got a "Never saw reconfiguration finish." message which appeared
 suspiciously quickly:

 {{{
 $ buildbot reconfig
 sending SIGHUP to process 4535
 [...] [-] beginning configuration update
 [...] [-] Loading configuration from [...]
 [...] [-] [a message my master.cfg "print"s]
 [...] [-] [a message my master.cfg "print"s]
 [...] [-] removing scheduler [...]
 [... 13 more messages of the above form ...]
 Never saw reconfiguration finish.
 }}}

 I checked and twistd.log got rotated in the middle of the reconfiguration
 - the messages that 'buildbot reconfig' output were now in twistd.log.1,
 and the rest of the output was in twistd.log:

 {{{
 [... 100 lines of other messages ...]
 [...] [-] Starting factory <buildbot.status.web.baseweb.RotateLogSite
 instance at 0x7f1cdc063320>
 [...] [-] Setting up http.log rotating 10 files of 10000000 bytes each
 [...] [-] configuration update complete
 }}}

 We have 53 schedulers, so lots of messages during reconfiguration, which I
 guess increases the opportunity to hit this bug.  I don't think I've ever
 seen it before though.

 Note that I'm using Buildbot from master at commit ID 9cd7f9a, which is a
 little over 2 years old.  I'm hoping to upgrade to 0.8.12 soon.

--
Ticket URL: <http://trac.buildbot.net/ticket/3360>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation