[Buildbot-devel] monitoring and tagging multiple git repositories

[Dustin J. Mitchell]

On Tue, Jun 7, 2011 at 6:44 AM, Dickon Reed <dickon at cantab.net> wrote:
> One option I am considering is to implement a change source that polls for
> untagged revisions across all branches in all repositories, and tags
> branches that have new revisions in all repositories. It would then invoke
> addChange with the revision being the name of the tag, and the comment
> containing the git log output for the new revisions introduced by that tag.
> There are some sketches along these lines
> at https://github.com/dickon/buildbot in src/master/buildbot/changes/multigit.py,
> which has unit test coverage. However, this is all pretty early on and I
> would be very interested to hear about any experiences people have had with
> this kind of approach.

So this sounds like a relatively new problem - there are an astounding
variety of crazy build systems out there, as it turns out.  Due to
that variety, I'm most interested in taking patches that add *highly*
generic support that can then be specialized for all of the
unimaginable build requirements our users find themselves with.

One generalization I've been toying with for a while is an "aggregate
sourcestamp", which would be a sourcestamp that includes revision
information for a whole tree of repositories.  In your case, your
ChangeSource would produce changes for single repositories, and a
specialized scheduler would use those changes to evolve a sourcestamp
that represents the latest revision in each repository, and then kick
off builds with that sourcestamp.

To give you an idea what we're up against, we also have users with an
unbounded set of repositories, where changes to an as-yet-unseen
repository should end up being built automatically.  That means that
the Source step can't be customized for any particular repository.
Others need to build each of a set of supporting libraries, and then
only build the final application against the most recent successful
version of each of those libraries.  Some want to only build the
supporting libraries on a nightly basis, and build the application
against binaries of that library.  Some have complex version
inter-dependencies, where the latest commits to project A can only be
built against versions of B prior to some revision R, but the
developers of B also want tests to run on the most recent commits to B
-- and A's developers need some way to indicate when A can build
against later versions of B.

Anyway, I'd love to have help fixing this problem - nobody's working
on a general approach to this, although we've got good support for
building multiple *unrelated* projects in the same master now, and
we've had a number of nipping-around-the-edges approaches to building
single projects from multiple repositories.

Dustin