[Buildbot-devel] Ideas on builds with multiple variants

[Greg Ward]

Here's the scenario: I've just setup a small BuildBot for testing a
Mercurial extension that I've written.  Obviously I want to make sure
that my extension continues to pass its tests with the current stable
version Mercurial, preferably on multiple platforms.  No surprise
there -- that's what BuildBot is for.

But it's more interesting than that: I also want to ensure that my
extension works with several versions of Mercurial.  The docs claim
that it supports Mercurial 1.3 and up, but any commit to my extension
could break that promise.  So what I want is something like

  for hgversion in (1.3 1.4 1.5); do
    test my extension with hg${hgversion}, assuming it's available on $PATH

(The idea is that build slave maintainers could install
/usr/local/bin/hg{1.3,1.4,1.5} as aliases for 1.3.x, 1.4.y, and 1.5.z,
the latest versions on each respective stable branch.)

But wait, it gets more fun: any commit to Mercurial itself could break
my extension.  This is most likely to happen on Mercurial's trunk (aka
"default" branch), which is going to become Mercurial 1.6.  So really
I should add

  cd <mercurial source tree>
  hg pull
  hg update
  make local
  test my extension with the hg just built

after the above loop.  Or as part of it.

I can think of lots of ways to implement this, but I can't see that
one way is obviously superior.  Possibilities:

  1) put the loop over $hgversion in master.cfg and require build
slave maintainers to set things up "just so" (probably OK because I
plan to be the only slave maintainer, but it still feels brittle)
  2) keep the loop over $hgversion in master.cfg, but put a script in
my source tree that takes $hgversion as an argument and ignores
missing Mercurial versions (same as #1 but a little less brittle)
  3) move the loop over Mercurial versions into my extension's source
tree, and just have BuildBot call that script
  4) keep a clone of Mercurial's source tree on the slave and use it
to build every required hg version for every build (requires two
source update steps, which is a bit weird but seems to work fine...
but it's a lot of wasted work, since Mercurial 1.3.1 is not going to
change)

Now before you think "bah, this is easy", here's my next thought: my
extension also claims to support Python 2.4 .. 2.x.  But any commit
could break that promise.  So really, I should *also* test with a
range of Python versions:

  for pyversion in 2.4.x 2.5.x 2.6.x trunk:
    for hgversion in 1.3.x 1.4.x 1.5.x trunk:
      build Mercurial $hgversion with Python $pyversion
      run my extension's tests with $hgversion and $pyversion

Now things get *really* fun: do I build a new Python on every slave
for every build?  Require multiple Python versions on $PATH?

Has anyone setup something similiar?  Ideas, thoughts, advice?  Thanks --

Greg