[Buildbot] #2833: probably deadlock situations with MasterLock

[Buildbot trac]

#2833: probably deadlock situations with MasterLock
----------------------+-----------------------
Reporter:  lunochod   |      Owner:
    Type:  undecided  |     Status:  new
Priority:  major      |  Milestone:  undecided
 Version:  0.8.9      |   Keywords:
----------------------+-----------------------
 Hi,

 I have the following constellation:

 Many builders build on two slaves which share a common hard disk.
 A SlaveLock exists that allows 2 resp. 4 parallel builds on each slave.

 The builders write (make install) and read (make, make test) libraries
 from a common directory.
 Further, builders trigger other builders that use the provided libraries.
 Sometimes builds fail because a library is linked when it is just written.

 Therefore I introduced a MasterLock 'harddisk_lock'. The 'make' and 'make
 test' ShellCommands have counting access, whereas the maximum number of
 accessing commands is 1000 which should be more than enough.
 The 'make install' commands have exclusive access.

 Now the strange behaviour:
 Sometimes there are only 'make' commands waiting for the harddisk lock and
 all builds stand still.
 The last two lines in twisted.log say:
 "aquireLocks ..."
 "step ... waiting for lock"
 After some minutes some lines in the logging say:
 "automatically retrying query after OperationalError (1.0s sleep)"
 followed by a
 "releaseLocks ..."

 From then on the builders are building as expected until the next "hang".
 These "hangs" are rather seldom but occure persistently.

 Sorry for this vague problem report. I don't know where to start to track
 this down further.

--
Ticket URL: <http://trac.buildbot.net/ticket/2833>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation