[Buildbot-devel] log file assertion error after master crashes hard

[Nathaniel Smith]

On Thu, Oct 06, 2005 at 01:59:36PM -0700, John Blair wrote:
> If the buildbot master crashes hard the log files are left in some sort
> of inconsistent state that causes new builds to throw an assertion,
> which causes a python exception after the buildbot master has been
> restarted.  I have cut & pasted the traceback below.  Builds can be
> resumed by deleting the master's log files.
> 
> I discovered this problem when our build server went down hard due to a
> power failure.  I am able to reproduce the problem by using 'kill -9' on
> the buildbot master.  That is, run "kill -9 $(cat twistd.pid)", then
> "make start".  Force a build, or check in some code, and you will see
> the exception.

We've observed this after power failures as well.  It wasn't
necessarily decremented by one, though; we saw it jump back like a
dozen numbers.  Pounding on the "force build" button repeatedly to
increment the number did get it working again eventually, but we had
some other confusing problems (maybe more power failures, maybe this
didn't quite fix it, I got confused) and I ended up wiping the master
dir and starting over.

-- Nathaniel

-- 
So let us espouse a less contested notion of truth and falsehood, even
if it is philosophically debatable (if we listen to philosophers, we
must debate everything, and there would be no end to the discussion).
  -- Serendipities, Umberto Eco