[Buildbot-devel] monotone and buildbot

[Brian Warner]

> I'm trying to set up a buildbot for monotone: http://venge.net/monotone/

Woot!

I wish you luck :)

> The basic challenge is that monotone makes some quite different
> assumptions from other revision control tools.  In particular, it has
> no concept of a branch that grows linearly over time, one change
> after another.  Instead, each checkin is represented by a SHA1
> describing its contents, and a branch is a DAG of these checkins.  So
> we have the concept of a branch "head", but -- there may be multiple
> heads.

Yeah. The place to add code is buildbot.process.step.Monotone(Source), and
the fun part will be computeSourceRevision(). This method is responsible for
taking a bunch of Change objects and deciding how to run the VC checkout
command. (also see Source.start to see how a forced build might override
that).

Hmm. The easiest thing to do is probably look for the latest (time-wise)
change, and then just build that. If you have multiple changes that don't
descend from one another which arrive close to each other, some of them will
lose (two "branches" under active development, someone commits to each at
about the same time). If there is an easy way for the buildmaster to tell
which change inherits/descends from which, it can rule out any that have
descendants in the same batch. That still may leave a set of changes which
ought to spawn multiple builds.

The real fix will probably be at a higher level, where the changes are handed
to the Builders and the accumulate them, waiting for the tree to become
"stable". Clearly this doesn't match Monotone's model.

Thomas and I were talking at PyCon about creating a more flexible way to
route Changes to various Builders, specifically to support multiple branches
in traditional VC systems like CVS and SVN. Those changes might make life
easier for something like Monotone as well.. ideally you'd want the Builders
to watch the changes coming in and separate them into related sets, so if I
commit 1->2 and 2->3 and 1->4, the Builder would know that 1->2+2->3 are one
path, and 1->4 is a different one. In this case, it should queue up two
Builds for running, one which will checkout 3 and another which will check
out 4. The place for this is in the Builder but I suspect there is some
support code that should be in place before it would be very clean.

The Builder's .buildable attribute should be changed to be a list of Build
objects that are all ready to go. This will be useful for more than
Monotone's non-linear concept of history: the 'try' feature will be able to
add multiple unrelated (specifically un-mergeable) Builds to this list. At
the moment there is only one slot for a build which is ready to go but is
waiting for the slave to become free, and any potential new build that would
get added to the list are instead merged into the pending one. The new
.buildable list would need some logic to look for a potentially-mergeable
Build, and would need to know that it's not allowed to merge new Changes into
a "try"-triggered Build. For Monotone, if a Change arrived that derived from
the revision that an pending Build was about to use, it would be allowed to
merge the Change into that Build. However a Change from some other lineage
would not: it would have to go into its own Build.

> Keep the linear display and everything, that works out okay, since
> development is overall generally linear and there are rarely more than 2-3
> parallel development lines, so just mushing them into a single linear
> timeline for display works.

Yeah, the vertical axis of that display is really time, even though it might
look like it's revision number. Internally the status code thinks of each
Builder as having a linear set of numbered Builds, but again it doesn't
assume that the order of those Builds has anything to do with the heritage of
the code inside them.

> It would sort of work for each builder to look at all newly arrived
> revisions, pick a head one, and build it; but then there would need to
> be some way to inform the changemaster that "actually, I only built
> half of those changes you gave me, so give me the other half again
> when I'm done", or something?

Yeah, the Builder needs to be changed to have multiple pending Builds,
otherwise it'd be a tangled mess.

> It also seems that forced builds need some way to specify which
> revision to build.  "current head" is neither well-defined, nor
> actually very useful; usually I want to know whether a particular tree
> I just committed broke anything on platform X, not whether some
> randomly chosen recent tree broke anything on platform X :-).

When you force a build, you should be able to give a specific revision
number. (this isn't implemented yet, but basically Builder.forceBuild()
should take a 'revision' argument that gets stuffed into
build.setSourceStamp). For Monotone this would just be the SHA-1 hash of the
revision/commit that you want it to use. But yeah, if you don't do that.. the
definition of "HEAD" is fuzzy. What do monotone developers do in practice? Is
there some sort of filtering criteria you give it when you join a project and
want to start working on the "latest"? Like "give me a revision that is
marked 'passes tests' that has the latest timestamp"? If we can convert that
same criteria into a value for the 'revision' argument to forceBuild, then we
can have the step.Monotone BuildStep do the same thing.

> Any thoughts on how to go about these things?  What part of
> buildbot's source code should I be looking at?  I don't understand the
> logic for generating and using changes very well yet;

Changes are created by the various ChangeSources (buildbot/changes/*.py) and
passed to the ChangeMaster, which then sends a copy into each active Builder
(through the BotMaster, I think, but I could be wrong). Each Builder has a
Build instance in its .waiting slot: all Changes are handed to it. That Build
decides if the Change is interesting or not, stuffs it into a list
(.importantChanges, .unimportantChanges) and starts a timer. When the timer
expires (meaning the tree has been stable for a while), the Build tells the
Builder that it's ready to go, and gets moved into the .buildable slot. When
the slave becomes available, it is moved to the .building slot and the build
process is actually started.

When the build process gets around to starting the Source step, that
BuildStep looks at its parent Build's list of Changes and derives a "source
stamp" from it, which is then passed to the slave for use in an -rFOO -style
argument.

> In the future, it would also be nice to take actions based on the
> results of builds -- for instance, monotone has the capability to mark
> in the repository whether a given revision works or not, and then make
> use of this information at 'update' time.  We also do various sorts of
> metrics tracking -- currently test coverage, hopefully performance
> benchmarks soon as well -- and it would be nice to tie these into the
> buildbot change trigger stuff instead of using the current ad hoc mess
> of shell scripts.  Where would I look to hook things like this in?

Yeah, I'd very much like to have those sorts of metrics. They should probably
be in Builder.buildFinished(), which should examine the Build that just
finished and send information about it to various places. The BuildStatus
object might be a better place: in that case the approach would be to have
the Build publish various data about itself to its related BuildStatus
object, then any BuildStatusReceiver instances which had subscribed to hear
about builds finishing could retrieve that status and do whatever they wanted
to with it (save it to disk, build graphs, etc).

Things that affect the build process or source control system should probably
be in Build or Builder, not BuildStatus. For example, if you want to tell
Monotone to mark the revision as "passing" once a build that includes that
revision has passed tests, that should probably be done from
Builder.buildFinished(). The main complication is that the buildmaster is not
guaranteed to be able to check out files or run other VC commands: in the
typical installation the slaves do all the VC work, the master just receives
Change notices.


whew, that was a mouthful. Hopefully it gives you something to start with.
Let me know how it goes!

cheers,
 -Brian