[Buildbot-commits] [SPAM] Re: [Buildbot] #664: sporadic connection lost problems

[Buildbot]

#664: sporadic connection lost problems
--------------------+-------------------------------------------------------
Reporter:  ddunbar  |        Owner:  ddunbar 
    Type:  defect   |       Status:  reopened
Priority:  major    |    Milestone:  0.7.12  
 Version:  0.7.12   |   Resolution:          
Keywords:           |  
--------------------+-------------------------------------------------------
Changes (by clown):

  * status:  closed => reopened
  * version:  => 0.7.12
  * resolution:  fixed =>


Comment:

 I've got a big build artifact directory: six 55Mb files - over 300 Mb.

 The DirectoryUpload can't upload it. On the slave:
 {{{
 2010/03/24 18:07 MSK [Broker,client]  startCommand:uploadDirectory [id 3]
 2010/03/24 18:07 MSK [Broker,client] SlaveDirectoryUploadCommand started
 2010/03/24 18:07 MSK [Broker,client] path: '...'
 }}}
 And nothing happens. Not even any CPU activity on the slave. Eventually,
 after a long (13 min in this case) on the slave:
 {{{
 2010/03/24 18:20 MSK [Broker,client] sendStatus {'header': 'sending ...'
 }
 2010/03/24 18:20 MSK [-] sending app-level keepalive
 2010/03/24 18:20 MSK [-] BotFactory.checkActivity: nothing from master for
 771 secs
 2010/03/24 18:20 MSK [-] SlaveFileUploadCommand._writeBlock():
 allowed=16384 readlen=16384
 2010/03/24 18:20 MSK [Broker,client] SlaveBuilder._ackFailed:
 SlaveBuilder.sendUpdate
 2010/03/24 18:20 MSK [Broker,client] BotFactory.keepaliveLost
 2010/03/24 18:20 MSK [Broker,client] finished: stderr=None, rc=0
 2010/03/24 18:20 MSK [Broker,client] sendStatus {'rc': 0}
 2010/03/24 18:20 MSK [Broker,client] SlaveBuilder.commandFailed
 <buildbot.slave.commands.SlaveDirectoryUploadCommand instance at
 0x86aa94c>
 2010/03/24 18:20 MSK [Broker,client] Traceback (most recent call last):
           File "/opt/csw/lib/python/twisted/internet/defer.py", line 307,
 in _runCallbacks
             self.result = callback(self.result, *args, **kw)
           File "/opt/csw/lib/python/site-
 packages/buildbot/slave/commands.py", line 903, in _err
             fire_when_done.errback(why)
           File "/opt/csw/lib/python/twisted/internet/defer.py", line 251,
 in errback
             self._startRunCallbacks(fail)
           File "/opt/csw/lib/python/twisted/internet/defer.py", line 294,
 in _startRunCallbacks
             self._runCallbacks()
         --- <exception caught here> ---
           File "/opt/csw/lib/python/twisted/internet/defer.py", line 307,
 in _runCallbacks
             self.result = callback(self.result, *args, **kw)
           File "/opt/csw/lib/python/site-
 packages/buildbot/slave/commands.py", line 1036, in finished
             self.sendStatus({'rc': self.rc})
           File "/opt/csw/lib/python/site-
 packages/buildbot/slave/commands.py", line 800, in sendStatus
             self.builder.sendUpdate(status)
           File "/opt/csw/lib/python/site-packages/buildbot/slave/bot.py",
 line 220, in sendUpdate
             d = self.remoteStep.callRemote("update", updates)
           File "/opt/csw/lib/python/twisted/spread/pb.py", line 382, in
 callRemote
             _name, args, kw)
           File "/opt/csw/lib/python/twisted/spread/pb.py", line 855, in
 _sendMessage
             raise DeadReferenceError("Calling Stale Broker")
         <class 'twisted.spread.pb.DeadReferenceError'>: Calling Stale
 Broker

 2010/03/24 18:20 MSK [Broker,client] Unhandled error in Deferred:
 2010/03/24 18:20 MSK [Broker,client] Traceback (most recent call last):
           File "/opt/csw/lib/python/twisted/internet/defer.py", line 307,
 in _runCallbacks
             self.result = callback(self.result, *args, **kw)
           File "/opt/csw/lib/python/site-
 packages/buildbot/slave/commands.py", line 903, in _err
             fire_when_done.errback(why)
           File "/opt/csw/lib/python/twisted/internet/defer.py", line 251,
 in errback
             self._startRunCallbacks(fail)
           File "/opt/csw/lib/python/twisted/internet/defer.py", line 294,
 in _startRunCallbacks
             self._runCallbacks()
         --- <exception caught here> ---
           File "/opt/csw/lib/python/twisted/internet/defer.py", line 307,
 in _runCallbacks
             self.result = callback(self.result, *args, **kw)
           File "/opt/csw/lib/python/site-packages/buildbot/slave/bot.py",
 line 253, in commandComplete
             d = self.remoteStep.callRemote("complete", failure)
           File "/opt/csw/lib/python/twisted/spread/pb.py", line 382, in
 callRemote
             _name, args, kw)
           File "/opt/csw/lib/python/twisted/spread/pb.py", line 855, in
 _sendMessage
             raise DeadReferenceError("Calling Stale Broker")
         <class 'twisted.spread.pb.DeadReferenceError'>: Calling Stale
 Broker
 }}}

 On master there is no target directory created. The corresponding
 buildstep is still 'Not finished', forewer. The corresponding builder is
 still 'building', forewer. Only master stop/start helps.

-- 
Ticket URL: <http://buildbot.net/trac/ticket/664#comment:6>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation