[Buildbot-commits] [Buildbot] #2455: buildbot silently reuses existing run #s

[Buildbot]

#2455: buildbot silently reuses existing run #s
--------------------+------------------------
Reporter:  virgilg  |       Owner:
    Type:  defect   |      Status:  new
Priority:  major    |   Milestone:  undecided
 Version:  0.8.7p1  |  Resolution:
Keywords:           |
--------------------+------------------------

Old description:

> After upgrading from 0.8.5 to 0.8.7p1 we see these:
>
> 2013-02-20 12:52:03-0800 [-] Warning: Overwriting old serialized Build at
> /Users/buildbot/buildmaster/tests-prerelease/5011-log-
> Deleting__Users_buildbot_buildslave_tests-prerelease_build_Logs.-stdio
>
> and our run #s are all messed up.
> It seems to happen immediately after a reconfigure.

New description:

 After upgrading from 0.8.5 to 0.8.7p1 we see these:

 {{{
 2013-02-20 12:52:03-0800 [-] Warning: Overwriting old serialized Build at
 /Users/buildbot/buildmaster/tests-prerelease/5011-log-
 Deleting__Users_buildbot_buildslave_tests-prerelease_build_Logs.-stdio
 }}}

 and our run #s are all messed up.
 It seems to happen immediately after a reconfigure.

--

Comment (by dustin):

 So, all of the bugs you've found are pointing to your having multiple
 buildmasters running in the same directory.

 In this case, there's only function which decides on the build number
 (`determineNextBuildNumber`).  Twisted is single-threaded, so that
 function can't race with itself in the same process.  So this could be
 caused by two processes trying to create build pickles in the same
 directory.

 Another alternative is running the buildmaster on a lame network
 filesystem where files do not appear immediately in the directory listing
 after they are created.

 Either option would explain duplicate files in the status_push queue, as
 well as the DB corruption.  The failure to receive signals might be caused
 by two masters running at once, but not by bad NFS.

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2455#comment:1>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation