merging builds (was Re: [Buildbot-devel] Accumulating Scheduler)

[Chris Leishman]

On Sun, Nov 27, 2005 at 08:32:07PM -0800, Brian Warner wrote:
<snip>
> The way the buildbot usually deals with this is by "merging" BuildRequests.
> Each Builder has a queue of pending BuildRequest objects.. one gets added to
> the queue each time the Scheduler fires, so there could be an arbitrarily
> large number of them stacked up, waiting for a buildslave to become
> available. However, once a slave *does* become available, the Builder walks
> through the entire queue and finds all the BuildRequests that can be merged
> with the first one. It then merges them all together, and starts a build that
> will resolve all of them. Each build can thus pop multiple BuildRequests off
> the queue, possibly all of them.
> 
> BuildRequests are mergeable if their SourceStamps are mergeable. Two
> SourceStamps are mergeable if they're both builds of HEAD (of the same
> branch), or if they're both builds of some set of Changes (all on the same
> branch). So hitting the "force build" button a zillion times in a row will
> result in a zillion BuildRequests, but only one (or maybe two) actual builds,
> since they'll all be merged together. Likewise, submitting a zillion changes
> (even if they're spaced out longer than the treeStableTimer, so they result
> in multiple BuildRequests) will only result in a finite number of builds,
> generally two: one for the first BuildRequest, and a second for all the rest
> of them that got submitted after the first one started.
> 
> When two BuildRequests (one with Changes 1,2,3 and a second with Changes
> 4,5,6) are merged, the resulting Build does a checkout of Change #6. It is
> possible that there was a problem with Change #2 and now the authors of
> Changes #4 #5 and #6 will be incorrectly blamed for it, but it's the only way
> to keep the BuildRequest queue from growing without bound. I picked this
> merge strategy to balance off timeliness of builds against total amount of
> work done for some set of Changes. (to be maximally fine-grained about
> assigning blame and to provide results as quickly as possible, you should
> build each Change separately; if you want to absolutely minimize the number
> of builds, you wait a couple days just in case they decide to commit
> something else).
> 
> The "14 pending builds" display is a bit scary and not entirely accurate. It
> would be equally correct to have the merge step take place when the new
> BuildRequest is *added* to the queue, rather than when one of them gets
> pulled off. Grep around for 'canBeMergedWith' in buildbot/process/builder.py
> to see where this logic is implemented.
> 
> So.. does that address the problem? If not, let me know, and I'll see if I
> can come up with answers to the second half of your message.

Thanks Brian, this makes a lot of sense.

So basically your saying that if 3 commits are all spaced out longer
than treeStableTimer, only two builds will result - as the first change
starts a build and then the second two are combined (as long as the
first build doesn't complete first).

Eg.

treeStableTimer = 10

00.00.00: commit 1

00.00.10: build submitted (build 1)
00.00.10: build starts (eta 10min)

00.01.00: commit 2

00.01.10: build submitted (build 2)

00.02.00: commit 3

00.02.10: build submitted (build 3)

00.10.10: build completes
00.10.10: build starts (with builds 2 & 3 merged)


Strangely this does not seem to be the behaviour I'm seeing.  How about
when a build is waiting for a lock?  Will all the commits made during
this time be merged?  Are changes submitted by svn_buildbot.py
mergeable?

I'll dig around in the code tomorrow and see if I can work it all out.

Thanks,
chris