[Buildbot-devel] Accumulating Scheduler

[Brian Warner]

> I'm trying to implement my own Scheduler (derived from
> buildbot.scheduler.Scheduler), that manages only a single builder and
> continues to accumulate changes after the treeStableTimer has expired
> while the builder is still busy.

I'll respond to the first half of your message right now.. then you tell me
if you still want an answer to the second half.

> The problem I'm having is that there can be quite a long window of time
> that a build gets deferred for (because there are no available slaves),
> and in that time people continue to commit at greater intervals than
> treeStableTimer.  Thus I can end up with very long queues of pending
> builds.

The way the buildbot usually deals with this is by "merging" BuildRequests.
Each Builder has a queue of pending BuildRequest objects.. one gets added to
the queue each time the Scheduler fires, so there could be an arbitrarily
large number of them stacked up, waiting for a buildslave to become
available. However, once a slave *does* become available, the Builder walks
through the entire queue and finds all the BuildRequests that can be merged
with the first one. It then merges them all together, and starts a build that
will resolve all of them. Each build can thus pop multiple BuildRequests off
the queue, possibly all of them.

BuildRequests are mergeable if their SourceStamps are mergeable. Two
SourceStamps are mergeable if they're both builds of HEAD (of the same
branch), or if they're both builds of some set of Changes (all on the same
branch). So hitting the "force build" button a zillion times in a row will
result in a zillion BuildRequests, but only one (or maybe two) actual builds,
since they'll all be merged together. Likewise, submitting a zillion changes
(even if they're spaced out longer than the treeStableTimer, so they result
in multiple BuildRequests) will only result in a finite number of builds,
generally two: one for the first BuildRequest, and a second for all the rest
of them that got submitted after the first one started.

When two BuildRequests (one with Changes 1,2,3 and a second with Changes
4,5,6) are merged, the resulting Build does a checkout of Change #6. It is
possible that there was a problem with Change #2 and now the authors of
Changes #4 #5 and #6 will be incorrectly blamed for it, but it's the only way
to keep the BuildRequest queue from growing without bound. I picked this
merge strategy to balance off timeliness of builds against total amount of
work done for some set of Changes. (to be maximally fine-grained about
assigning blame and to provide results as quickly as possible, you should
build each Change separately; if you want to absolutely minimize the number
of builds, you wait a couple days just in case they decide to commit
something else).

The "14 pending builds" display is a bit scary and not entirely accurate. It
would be equally correct to have the merge step take place when the new
BuildRequest is *added* to the queue, rather than when one of them gets
pulled off. Grep around for 'canBeMergedWith' in buildbot/process/builder.py
to see where this logic is implemented.

So.. does that address the problem? If not, let me know, and I'll see if I
can come up with answers to the second half of your message.

cheers,
 -Brian