[Buildbot-devel] Building from multiple sources

[Brian Warner]

> Is it possible to build from multiple sources with buildbot?

As Alexander pointed out, it's kind of a kludge. There's been a lot of demand
for a feature like this recently, so I'm eager to find a clean solution. In
particular, Glyph started a thread on the python-dev mailing list about
creating a set of community buildbots that would help projects like Twisted
discover incompatibilities with development versions of other projects [like
python] upon which they depend. With the Schedulers and remote-status
interfaces that are already in place, we could have the Twisted buildbot
watch the Python buildbot and fire off a Twisted build every time a new
Python checkin passed all the python tests.

There are two issues. The first (and easier to work around) is mechanics. The
second is status reporting.

Setting workdir= on each step is how you make any given command run in a
particular directory. The buildbot is optimized for making one-source-tree
projects simple to build, which is why workdir= has a default value, and why
all of the Source steps arrange to put the checked out source code into that
same subdirectory. The simplest way to approach a multiple-source-tree
project is to have multiple working directories, do a separate checkout of
each tree (within a single Build), do a build of the dependencies (the
"upstream" libraries that are needed to build or test the "downstream"
project) in one workdir, then do a build of the dependent project (in a
second workdir) in a way that uses the results of the upstream build.

One barrier here is that the Source checkout steps weren't designed with
multiple source trees in mind. You can override workdir= to control where the
source code winds up, but the most useful checkout mode (mode='copy') is
hardwired to put the unmodified source tree in a directory named "source",
and using multiple Source steps (for unrelated source trees) will collide.
(note that other modes, like "clobber" or "export", do not use this hardwired
directory). A second barrier is that the Source step is designed to get a
"SourceStamp" from the build, which indicates what (singular) revision
describes this build. To use multiple Source steps, you'd have to set
alwaysUseLatest=True on the step that checks out the "upstream" libraries, so
you could reserve the Build's sourcestamp for the "downstream" project's
tree.

Having the upstream build take place in a completely separate Builder
(running on the same slave as your downstream Builder) could work too, except
that the Builders are nominally independent and you have to violate that
abstraction boundary to get one of them to point at the compiled
libraries/etc of the other. Doing it this way would probably make it easier
to leave the compiled upstream tree in place, and use it for all builds of
the downstream tree. You could probably rig this with two separate
Schedulers: one of them should only watch the upstream code, and trigger a
local build of the upstream tree (using a Source checkout step with
alwaysUseLatest=True). The second should watch the downstream code, and
trigger the downstream build (which should use CFLAGS=-I or PYTHONPATH= or
whatever to use the upstream libraries). You'd want to have a lock between
the two builds, to make sure you don't run your downstream build while the
upstream one is half-compiled.

Doing this all cleanly (including status reporting) is even thornier. I think
that the current SourceStamp that each build contains should perhaps become a
dictionary (which maps sub-project name to each SourceStamp).. that way you
could tell each Source step which sub-project it is responsible for, and it
could extract the given SourceStamp from the dictionary. We'd add a new kind
of Scheduler that knows about multiple sub-projects and remembers the most
recent revision for each one, spawning BuildRequests that use the most recent
code from each of the subprojects. The Change objects would be modified to
include a sub-project name, so the Scheduler could tell them apart. We'd
definitely need a better web page, since you'd be adding an extra dimension
to them (at the moment you can think of the display as builder * revision,
with sub-projects that would be [builder * upstream_revision *
downstream_revision]).

So anyways, those are some starting points. I'm definitely interested in
making this sort of thing possible, as long as we can find a way to keep the
simplicity of single-project buildbots intact.

cheers,
 -Brian