[Buildbot-devel] Changes, ChangeSources, and SourceStamps

[Dustin J. Mitchell]

I've been puzzling over this a bit, and now I'm going to post the
results in hopes that (a) this will prove useful to someone sometime
in the future, (b) y'all will correct me if I'm wrong.  This might
make a good Wiki page, too.

A ChangeSource watches some source-code repository, waiting for things
to change.  Some ChangeSources poll, while others arrange to be
notified of new changes.  Either way, when something in the repository
changes, the ChangeSource pops out some Change objects and sends them
to the BuildMaster, which sends them on to each configured Scheduler.

Not all schedulers respond to changes, but those which do queue up the
Change and (re)start their build-stable timer.  While the timer runs,
more changes may come in.  When the timer expires, the Scheduler
creates a SourceStamp with all of the accumulated Changes.  The
resulting SourceStamp has no revision or patch, but its branch is
based on the changes.  It sticks that SourceStamp into a BuildSet
along with the list of targeted builders, and submits it to the
BuildMaster.  The BuildMaster compiles a list of Builder objects, and
passes that to the BuildSet's start().

The BuildSet makes a BuildRequest for each builder, and submits it to
the corresponding builder.  The builder adds the request to its list
of buildable requests, and tries to start a build.  Beginning with the
earliest request, it merges requests until it runs out of requests or
finds an incompatible request.  A request can be merged if its
SourceStamps can.  SourceStamps can be merged if they are both based
on a set of Changes, rather than a specific revision, or if they both
specify the same revision.  Merging SourceStamps merely concatenates
their list of changes.

After that, a Build is set up based on the BuildRequest's (perhaps
merged) SourceStamp.  The Source step does *not* look at the changes,
however -- it uses the branch (which was determined from the changes,
above), and checks out the most recent source.  This may, in fact,
include source-code changes not in the list of Changes, if they were
committed between the last poll by the ChangeSource and the submission
of the BuildRequest.  Different builders may also build different
revisions, again based on their different start times.  Only after the
Source step is finished do we find out what revision was actually
checked out (got_revision).

Builds can, of course, be triggered in other ways -- through Try,
TriggerableSchedulers, forced builds, etc. -- and the resulting
SourceStamps may have a revision, patch, and/or branch, but no
Changes.

Dustin

-- 
Storage Software Engineer
http://www.zmanda.com