[Buildbot-devel] Serializing builds

[Brian Warner]

> For performance reasons, the nightly builds can't occur simultaneously. 
> As far as I can see, BuildBot doesn't have a mechanism to serially 
> trigger a sequence of builds.
> 
> Will the PBChangeSource do what I need? What's the best way to 
> accomplish this?

The best approach is probably to write a module to start one build, wait for
it to finish, start the next one, etc. The IBuilderStatus and IBuilderControl
interfaces have most of what you need (see buildbot/interfaces.py for
details). You'd want something like:

class NightlySequentialBuilds(twisted.application.service.Service):
    def start(self):
        self.builders = ["one", "two", "three"]
        self.next()

    def next(self, res=None):
        if not self.builders:
            return
        name = self.builders.pop(0)
        builder = IControl(master).getBuilder(name)
        buildcontrol = builder.forceBuild(None, "nightly build")
        buildstatus = buildcontrol.getStatus()
        d = buildstatus.waitUntilFinished()
        d.addCallback(self.next)

If you wanted the build to only run if the previous build had succeeded, you
could check for 'res.getResults() != FAILURE' at the start of next().

This module would need a reference to the BuildMaster object. The easiest way
to get that is to add it as a status-target (i.e. include it in c['status']
in master.cfg), because all status targets are service-children of the
BuildMaster, so their .parent attribute points to the BuildMaster:

    def next(self, res=None):
        master = self.parent
        ...


It would also be possible to accomplish most of the same goal by simply using
the PB debug interface (contrib/debugclient.py) to force a build at certain
times. You wouldn't get a guarantee that the builds would be non-overlapping,
however, you'd just have to schedule the triggers far enough apart to avoid
overlap.

Using the PBChangeSource would trigger all the builds to run at once. Rather,
it would put all the builds into the "waiting" state at the same time: you
could adjust the timeouts separately and try to keep the builds from
overlapping, but that would suffer from the same uncertainty as driving it
through the debug interface.

[elliot]
> I'm wondering if the interlocks feature will help with this.

Only partially. "Interlock" is unfortunately the wrong name, that feature
should really be named "Dependency" (and I plan to fix it RSN). You could set
up builder C to depend upon B, and B upon A, and then fake a Change to
trigger them all. B and C would be blocked by the Dependency(/Interlock), so
only A would run. If A finished successfully, B would run. If B passes, C
will run.

Dependencies(/Interlocks) are meant for situations where one build must run
*successfully* to trigger another, like in the Twisted case where we don't
even bother building .deb packages unless all the unit tests are passing. The
'debuild' Builder has a Dependency(/Interlock) on the full-2.2 and full-2.4
Builders.


There is a separate feature on the TODO list named "Lock", which would
provide just this sort of one-build-at-a-time exclusivity, regardless of
whether the builds pass or fail. This is for both performance reasons (the
Twisted buildslave that handles about half of the Builders, for example, gets
really bogged down when there are four tests running at once, so it'd be nice
to be able to limit that a bit), and because sometimes tests involve
resources that really cannot be shared (the Twisted tests used to use
fixed-numbered listening ports, before we fixed that, so running two test
suites at the same time could cause port-in-use collisions and spurious
failures).

Hopefully I'll get some time to implement Lock soon (although patches are
always welcome, hint hint :). For now, if your builds almost always succeed,
you can use Interlock(/Dependency) and get fairly similar results.


cheers,
 -Brian