[Buildbot-devel] log file assertion error after master crashes hard

[John Blair]

If the buildbot master crashes hard the log files are left in some sort
of inconsistent state that causes new builds to throw an assertion,
which causes a python exception after the buildbot master has been
restarted.  I have cut & pasted the traceback below.  Builds can be
resumed by deleting the master's log files.

I discovered this problem when our build server went down hard due to a
power failure.  I am able to reproduce the problem by using 'kill -9' on
the buildbot master.  That is, run "kill -9 $(cat twistd.pid)", then
"make start".  Force a build, or check in some code, and you will see
the exception.

I haven't been able to trace the source of this problem myself yet, but
I think I have one clue.  It appears that the "build counter" is
decremented by one after the buildbot master is restarted.  That is, if
a builder's last build is number 7, after buildbot is restarted it will
think the last build is number 6.  The build logs for build 7 are still
in the build/ directory, but they won't be shown in the waterfall
display.

Thanks for any help.  I'd like to get rid of this so I can count on
buildbot to restart un-attended w/o resorting to blowing away the build
logs.

Here's the traceback to the assertion error:

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/buildbot/process/builder.py", line 364, in startBuild
    d = build.startBuild(bs, self.expectations, self.remote)
  File "/usr/lib/python2.3/site-packages/buildbot/process/base.py", line 311, in startBuild
    self.startNextStep()
  File "/usr/lib/python2.3/site-packages/buildbot/process/base.py", line 392, in startNextStep
    d = defer.maybeDeferred(s.startStep, self.remote)
  File "/usr/lib/python2.3/site-packages/twisted/internet/defer.py", line 129, in maybeDeferred
    result = f(*args, **kw)
--- <exception caught here> ---
  File "/usr/lib/python2.3/site-packages/buildbot/process/step.py", line 475, in startStep
    skip = self.start()
  File "/usr/lib/python2.3/site-packages/buildbot/process/step.py", line 987, in start
    self.startVC()
  File "/usr/lib/python2.3/site-packages/buildbot/process/step.py", line 1179, in startVC
    ShellCommand.start(self, errorMessage)
  File "/usr/lib/python2.3/site-packages/buildbot/process/step.py", line 683, in start
    loog = self.addLog("log")
  File "/usr/lib/python2.3/site-packages/buildbot/process/step.py", line 577, in addLog
    loog = self.step_status.addLog(name)
  File "/usr/lib/python2.3/site-packages/buildbot/status/builder.py", line 706, in addLog
    log = LogFile(self, name, logfilename)
  File "/usr/lib/python2.3/site-packages/buildbot/status/builder.py", line 218, in __init__
    assert not os.path.exists(self.getFilename())
exceptions.AssertionError: