[Buildbot-commits] [Buildbot] #2395: test_master fails randomly

[Buildbot]

#2395: test_master fails randomly
-------------------+--------------------
Reporter:  dustin  |      Owner:  dustin
    Type:  defect  |     Status:  new
Priority:  major   |  Milestone:  0.8.8
 Version:  master  |   Keywords:
-------------------+--------------------
 We "occasionally" (approximately one builder per commit) see error pairs
 like this:
 {{{
 buildbot.test.integration.test_master.RunMaster.test_master2 ... Exception
 twisted.internet.defer._DefGen_Return: _DefGen_Return() in <generator
 object _sortBuilders at 0x35374b0> ignored
 [ERROR][ERROR]

 ===============================================================================
 [ERROR]
 Traceback (most recent call last):
   File
 "/home/dustin/code/buildbot/t/buildbot/master/buildbot/process/botmaster.py",
 line 385, in maybeStartBuildsOn
     yield self._sortBuilders(list(existing_pending | new_builders))
 exceptions.GeneratorExit:

 buildbot.test.integration.test_master.RunMaster.test_master2
 ===============================================================================
 [ERROR]
 Traceback (most recent call last):
   File
 "/home/dustin/code/buildbot/t/buildbot/master/buildbot/process/botmaster.py",
 line 444, in _sortBuilders
     builders = yield defer.maybeDeferred(lambda :
 exceptions.GeneratorExit:

 buildbot.test.integration.test_master.RunMaster.test_master2
 -------------------------------------------------------------------------------
 Ran 2 tests in 0.612s
 }}}

 They all seem to occur on Python 2.7, but there's a small sample size, so
 that may be a false lead.

 The error occurs in different tests, but with exactly the same tracebacks:
 {{{
 buildbot.test.integration.test_upgrade.TestWeirdChanges.testUpgradeChangeNoRevision
 buildbot.test.regressions.test_import_unicode_changes.TestUnicodeChanges.testUTF16Change
 }}}

 These both run after test_master.  This suggests there's some "lingering"
 after-effect of test_master, and somehow trial is missing it and it's
 managing to inject a very weird failure into the reactor later.

 I'd suspect threads, but the test is configured to shut down the
 threadpool, and according to {{{threading.enumerate}}} it's doing so
 successfully.

 I can't replicate this problem on my machine.  I ran about 45,000 reps
 overnight with no ill effects.

 The test is skipped for now.

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2395>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation