[Buildbot-commits] [Buildbot] #1881: buildbot throws lots of Exceptions when a Build's logs disappear before it finishes

[Buildbot]

#1881: buildbot throws lots of Exceptions when a Build's logs disappear before it
finishes
----------------------+-----------------------
Reporter:  bhearsum   |      Owner:
    Type:  undecided  |     Status:  new
Priority:  major      |  Milestone:  undecided
 Version:  0.8.3p1    |   Keywords:
----------------------+-----------------------
 We see this case whenever a slave gets hung mid-job and we don't notice
 until after a something has cleaned up its master-side logs. There's a few
 Exceptions I've seen like this. Here's a couple:
         Traceback (most recent call last):
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/twisted/internet/defer.py", line 249, in addCallbacks
             self._runCallbacks()
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/twisted/internet/defer.py", line 441, in _runCallbacks
             self.result = callback(self.result, *args, **kw)
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/twisted/internet/defer.py", line 318, in callback
             self._startRunCallbacks(result)
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
             self._runCallbacks()
         --- <exception caught here> ---
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/twisted/internet/defer.py", line 441, in _runCallbacks
             self.result = callback(self.result, *args, **kw)
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py",
 line 1064, in <lambda>
             d.addCallback(lambda res:
 self.createSummary(cmd.logs['stdio']))
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/buildbotcustom/steps/unittest.py", line 808, in createSummary

           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/buildbotcustom/steps/unittest.py", line 144, in
 summarizeLogJetpacktests
             for line in log.readlines():
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
 line 400, in readlines
             alltext = "".join(self.getChunks([channel], onlyText=True))
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
 line 339, in getChunks
             f = self.getFile()
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
 line 317, in getFile
             return open(self.getFilename(), "r")
         exceptions.IOError: [Errno 2] No such file or directory:
 './mozilla-central_leopard-o-debug_test-jetpack/343-log-jetpack-stdio'



 2011-03-13 19:51:48-0700 [-] Unhandled Error
         Traceback (most recent call last):
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/twisted/internet/defer.py", line 441, in _runCallbacks
             self.result = callback(self.result, *args, **kw)
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/twisted/internet/defer.py", line 318, in callback
             self._startRunCallbacks(result)
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
             self._runCallbacks()
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/twisted/internet/defer.py", line 441, in _runCallbacks
             self.result = callback(self.result, *args, **kw)
         --- <exception caught here> ---
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py",
 line 885, in failed
             self.step_status.stepFinished(EXCEPTION)
           File "/builds/buildbot/tests_master6/lib/python2.6/site-
 packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
 line 1047, in stepFinished
             if os.path.getsize(loog.getFilename()) > logCompressionLimit:
           File
 "/builds/buildbot/tests_master6/lib/python2.6/genericpath.py", line 49, in
 getsize
             return os.stat(filename).st_size
         exceptions.OSError: [Errno 2] No such file or directory:
 './mozilla-central_leopard-o-debug_test-jetpack/343-log-jetpack-stdio'



 These sort of things need to get reported somewhere, but raw Exceptions in
 the twistd.log don't seem like the right place. Ideally, errors like this
 should make their way to Status plugins, but I don't think that's possible
 yet. For now, maybe it's best to report these in a nicer way in
 twistd.log.

-- 
Ticket URL: <http://trac.buildbot.net/ticket/1881>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation