[Buildbot-devel] multiple readers/single writer lock

[A.T.Hofkamp]

Hello all,

In my setup, several .tar.gz files are built from one svn working copy. Each 
has its own builder. To prevent many equal svn working copies, I have made a 
small 'working-copy' management script that manages several working copies in 
a branch/revision directory structure.

Each builder queries a particular branch/revision combination, and the script 
adds that to the directory structure if it doesn't exist (and else, does nothing).

There is also a builder that removes old working copies.

(The master and all slaves run at the same machine, so they can exchange data 
via the file system.)


At the moment, I have an exclusive master access lock on the working copies 
for each builder, so the same branch/revision combination is never queried 
twice concurrently, and also to prevent removal of a working copy by the 
cleanup builder while a .tar.gz file is being constructed.


This is working quite nicely, but not very fast. The exclusive access forms a 
bottle-neck in the builds.

However, since the working copies are not modified by the build process, I'd 
like to have several .tar.gz builds being executed concurrently, sharing 
working-copies. However, the cleanup builder should only run when no other 
builder is active, in other words, I need a multiple readers - single writer lock.


Can this be done in 0.7.7 ?

I looked at the maxCount property of the locks, and they seem to be useful 
only for limiting the number of builders/slaves on a per-builder/per-slave 
basis, while I'd like to have intra-builder exclusion locking.
(ie "maximal 3 builders with name 'x' AND 5 builders with name 'y'" rather 
than "1 cleanup builder OR 2 (in my case) .tar.gz builders" which I'd like to 
have).

Is my interpretation of the maxCount property correct?

If no, I'd be very happy :) , and the remainder of this post is unneeded.

If yes, I can have a go at improving the situation.

I have considered an alternative interpretation of the count towards 'weight'. 
A lock can take at most x units weight. Each builder adds some weight, thus 
limiting the total number of builders. While this would solve my problem, I 
fear it also makes my interpretation of the existing semantics impossible.

A solution would be to add the weight concept to the current semantics. Then 
both ideas can be expressed. To me, it doesn't seem a clean solution, too many 
different numbers.

How does the list consider this?



Albert