[Buildbot-commits] [Buildbot] #1881: buildbot throws lots of Exceptions when a Build's logs disappear before it finishes
Buildbot
nobody at buildbot.net
Tue Mar 15 14:54:35 UTC 2011
#1881: buildbot throws lots of Exceptions when a Build's logs disappear before it
finishes
---------------------+--------------------
Reporter: bhearsum | Owner:
Type: defect | Status: new
Priority: major | Milestone: 0.8.4
Version: 0.8.3p1 | Resolution:
Keywords: |
---------------------+--------------------
Changes (by dustin):
* type: undecided => defect
* milestone: undecided => 0.8.4
Old description:
> We see this case whenever a slave gets hung mid-job and we don't notice
> until after a something has cleaned up its master-side logs. There's a
> few Exceptions I've seen like this. Here's a couple:
> Traceback (most recent call last):
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/twisted/internet/defer.py", line 249, in addCallbacks
> self._runCallbacks()
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/twisted/internet/defer.py", line 441, in _runCallbacks
> self.result = callback(self.result, *args, **kw)
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/twisted/internet/defer.py", line 318, in callback
> self._startRunCallbacks(result)
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
> self._runCallbacks()
> --- <exception caught here> ---
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/twisted/internet/defer.py", line 441, in _runCallbacks
> self.result = callback(self.result, *args, **kw)
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py",
> line 1064, in <lambda>
> d.addCallback(lambda res:
> self.createSummary(cmd.logs['stdio']))
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/buildbotcustom/steps/unittest.py", line 808, in createSummary
>
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/buildbotcustom/steps/unittest.py", line 144, in
> summarizeLogJetpacktests
> for line in log.readlines():
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
> line 400, in readlines
> alltext = "".join(self.getChunks([channel], onlyText=True))
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
> line 339, in getChunks
> f = self.getFile()
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
> line 317, in getFile
> return open(self.getFilename(), "r")
> exceptions.IOError: [Errno 2] No such file or directory:
> './mozilla-central_leopard-o-debug_test-jetpack/343-log-jetpack-stdio'
>
>
> 2011-03-13 19:51:48-0700 [-] Unhandled Error
> Traceback (most recent call last):
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/twisted/internet/defer.py", line 441, in _runCallbacks
> self.result = callback(self.result, *args, **kw)
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/twisted/internet/defer.py", line 318, in callback
> self._startRunCallbacks(result)
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
> self._runCallbacks()
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/twisted/internet/defer.py", line 441, in _runCallbacks
> self.result = callback(self.result, *args, **kw)
> --- <exception caught here> ---
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py",
> line 885, in failed
> self.step_status.stepFinished(EXCEPTION)
> File "/builds/buildbot/tests_master6/lib/python2.6/site-
> packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
> line 1047, in stepFinished
> if os.path.getsize(loog.getFilename()) > logCompressionLimit:
> File
> "/builds/buildbot/tests_master6/lib/python2.6/genericpath.py", line 49,
> in getsize
> return os.stat(filename).st_size
> exceptions.OSError: [Errno 2] No such file or directory:
> './mozilla-central_leopard-o-debug_test-jetpack/343-log-jetpack-stdio'
>
>
> These sort of things need to get reported somewhere, but raw Exceptions
> in the twistd.log don't seem like the right place. Ideally, errors like
> this should make their way to Status plugins, but I don't think that's
> possible yet. For now, maybe it's best to report these in a nicer way in
> twistd.log.
New description:
We see this case whenever a slave gets hung mid-job and we don't notice
until after a something has cleaned up its master-side logs. There's a few
Exceptions I've seen like this. Here's a couple:
{{{
Traceback (most recent call last):
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/twisted/internet/defer.py", line 249, in addCallbacks
self._runCallbacks()
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/twisted/internet/defer.py", line 441, in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/twisted/internet/defer.py", line 318, in callback
self._startRunCallbacks(result)
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/twisted/internet/defer.py", line 441, in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py",
line 1064, in <lambda>
d.addCallback(lambda res:
self.createSummary(cmd.logs['stdio']))
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/buildbotcustom/steps/unittest.py", line 808, in createSummary
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/buildbotcustom/steps/unittest.py", line 144, in
summarizeLogJetpacktests
for line in log.readlines():
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
line 400, in readlines
alltext = "".join(self.getChunks([channel], onlyText=True))
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
line 339, in getChunks
f = self.getFile()
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
line 317, in getFile
return open(self.getFilename(), "r")
exceptions.IOError: [Errno 2] No such file or directory:
'./mozilla-central_leopard-o-debug_test-jetpack/343-log-jetpack-stdio'
}}}
{{{
2011-03-13 19:51:48-0700 [-] Unhandled Error
Traceback (most recent call last):
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/twisted/internet/defer.py", line 441, in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/twisted/internet/defer.py", line 318, in callback
self._startRunCallbacks(result)
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
self._runCallbacks()
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/twisted/internet/defer.py", line 441, in _runCallbacks
self.result = callback(self.result, *args, **kw)
--- <exception caught here> ---
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py",
line 885, in failed
self.step_status.stepFinished(EXCEPTION)
File "/builds/buildbot/tests_master6/lib/python2.6/site-
packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py",
line 1047, in stepFinished
if os.path.getsize(loog.getFilename()) > logCompressionLimit:
File
"/builds/buildbot/tests_master6/lib/python2.6/genericpath.py", line 49, in
getsize
return os.stat(filename).st_size
exceptions.OSError: [Errno 2] No such file or directory:
'./mozilla-central_leopard-o-debug_test-jetpack/343-log-jetpack-stdio'
}}}
These sort of things need to get reported somewhere, but raw Exceptions in
the twistd.log don't seem like the right place. Ideally, errors like this
should make their way to Status plugins, but I don't think that's possible
yet. For now, maybe it's best to report these in a nicer way in
twistd.log.
--
--
Ticket URL: <http://trac.buildbot.net/ticket/1881#comment:1>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation
More information about the Commits
mailing list