[Buildbot-commits] [Buildbot] #2395: test_master fails randomly

[Buildbot]

#2395: test_master fails randomly
-------------------+---------------------
Reporter:  dustin  |       Owner:  dustin
    Type:  defect  |      Status:  closed
Priority:  major   |   Milestone:  0.8.8
 Version:  master  |  Resolution:  fixed
Keywords:          |
-------------------+---------------------

Comment (by dustin):

 So, to summarize for others who might run into a very twist(y|ed) problem
 like this in the future:

 The `maybeStartBuild` method returned a Deferred, because it operated as a
 chain of asynchronous operations, but none of its callers really cared
 about that Deferred.  They did the usual `d.addErrback(log.err)`, just to
 be safe.

 Depending on unpredictable timing issues, sometimes a test would end while
 an invocation of this method was still processing.  In particular, it was
 waiting for a DB operation that was never going to occur, because the DB
 pool had been shut down.  This evaded the notice of trial's checks for a
 dirty reactor, because the frozen state was in a queue for the DB pool,
 rather than in an outstanding `defer.callLater` or an open TCP port.

 Ordinarily, an unfinished Deferred wouldn't be a problem -- it will
 eventually get gc'd and forgotten.  However, due to heavy use of
 `inlineCallbacks`, this Deferred has a generator attached to it.  When a
 generator is gc'd, the gc calls `g.close()`, which raises a
 `GeneratorExit` exception in the generator.  Generators aren't supposed to
 catch these, but the two methods in question here -- `_sortBuilders` and
 `maybeStartBuildsOn` -- use bare try/except blocks and thus do catch and
 log the `GeneratorExit` exception.

 The Python gc notices this and prints the warning message about the
 generator ignoring `GeneratorExit`.  But the `GeneratorExit` was logged as
 part of handling it, and trial treats logged errors as `[ERROR]` in
 whatever test is currently running.

 So, the fix was not to leave those Deferred operations running, but now
 that I've written this, I should also change the bare try/except's to use
 'except Exception'.

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2395#comment:10>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation