[Buildbot-devel] scheduler questions

[Joe MacDonald]

On 3/11/06, Marnix Klooster <marnix.klooster at gmail.com> wrote:
>
> Stefan Seefeld wrote:
>
> > Brian Warner wrote:
> >
> >>> On a related note, is there anything I can do to control the level of
> >>> concurrency, i.e. the number of processes run in parallel on a given
> >>> machine (e.g. slave) ? It would be good to be able to constrain the
> >>> number
> >>> of builds executed in parallel on a particular machine, to avoid
> >>> excessive
> >>> high loads.
> >>
> >>
> >>
> >> We have slave-wide Locks, which can keep the number of processes down
> >> to one
> >> per slave. With a little bit of work, these could probably be
> >> enhanced to
> >> behave more like multiple-holder semaphores, so you could have two or
> >> three
> >> Steps or Builds running at the same time on any given slave.
> >
> >
> > Yes, I think that would be very useful indeed.
>
>
> I second that.
>
> Practical use case that could be implemented by this mechanism (I
> think): I've got a pool of resources (in this case, for example, 4
> ClearCase views with common derived object storage), and a bunch of
> builds (some 25) for which I don't want to create 25 corresponding views
> (because the storage would be too much).  So when a build starts, I want
> it to take a free ClearCase view from the pool of 4, use it during the
> build, and then free it up again for other builds.


If anyone is interested, I'd implemented a multiple-reader, single-writer
subclass of the master locks for a very similar purpose about six months
ago.  By the time my employer worked out the logistics of how I could submit
code back to the project I had already fallen behind the curve (this is in
0.7.0)  and I've been just sitting on the code ever since.  I can share it
now, though, so I'd be happy to clean it up a little bit and provide a patch
for anyone who wanted to try it out.  Alternatively, I can check to see how
it plays with the latest version in case it would actually be able to be
incorporated.

The way it works now is this.

if (requesting read access lock) and ((lock is free) or (lock is held by
another reader))
  you get the lock and proceed

if (requesting read lock) and (lock is held by a writer)
  you block until the writier is done of the lock in the normal way

if (freeing a read lock) and (others are still holding the read lock)
  you finish but the lock is still held in read mode

if ((freeing a read lock) and (no other readers)) or (freeing a write lock)
  lock is completely unlocked

That's the very high level behaviour I modelled in the class.  Of course it
handles all the corner cases I could think of (ensuring readers don't starve
writers, for example) and it also supports allowing a number of 'writers'
larger than 1 if desired, since I'd actually wanted a way to allow four
writers and an unspecified number of readers.  It's all based on the
Master-wide locks, though, so it won't be a drop-in solution for the OP, but
it's probably a step in the right direction.  Interestingly enough, though,
my requirement sounds very much like the CC situation above, but it's not
that one at all.  :-)

-J.


--
-Joe.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20060312/de083769/attachment.html>