[Buildbot-devel] massive amounts of build variants, i.e., Firefox localizations repackagings

[Axel Hecht]

Hi,

short intro to me, I'm Axel Hecht, l10n coordinator with Mozilla.
Read, I'm not sharing an office with Rob Campbell, but an irc channel,
or 4.

I'm looking into using buildbot for helping us in repackaging our
localized versions of Firefox. That puts us into:

- a few dozen localizations (targeting 100+) for each product
(Firefox, Thunderbird (lower number)), for each release branch
(currently 1.5.0.x and 2.0.0.x).
- three platforms (win, mac, linux)
-  one tagged source

The repacks are essentially using one binary and two source trees
(en-US and localizations) and do some repackaging of the binary with
data from the source trees.

I blogged about an initial stab at this at
http://blog.mozilla.com/axel/2007/04/30/the-incubator/.

What I'd like to be able to do is:

- throw hardware at the problem.
- not waste resources.

We're currently pretty good on the second, without a chance at the first.

The first problem I have is getting a source onto the slave. On the
incubator test stage, I'm using a regular scheduler (in my mind, per
platform) to get the source on each slave (currently one), and then
use dependent schedulers for each locale/application/platform
combination. AFAICT, that would block the complete build on one of the
platform slaves dying.

My current thinking is, I don't want to repack a hundred localizations
at a time, so doing slave locks for the repackaging phase sounds like
a good idea in the first place. If I do that, in theory it sounds
close to use the same lock for the source checkout, too. Yes, I don't
really have a problem with 100 builddirs, but checking out a 100
source dirs is something that I'd rather avoid. So if I could set the
maxCount for the check-out step to 1, and to, say, some single digit
number for the repackaging, I'd expect to see all localizations to get
repackaged for a particular buildrequest, and then the lock would be
free for the next check-out on that slave.

Does hacking on the slave locks to enable them to have step-dependent
maxCounts sound like something reasonable, or am I making a badly
tuned gedanken experiment here and should do it differently?

The other thing I want to do is, we're moving parts of our build and
test infrastructure into python. Partly, because I (and others) were
thinking "cool, we can easily hook that up to buildbot". Seems there
is no free lunch, getting custom code up in the slave is unsupported,
AFAICT. Is that right? I don't see a way to get to
registerSlaveCommand, at least. I'm currently writing a patch to
slave/commands.py and steps/shell.py to enable independent
ISlaveCommand implementations, hooking up a slave command that uses
the imp module from __new__ to return arbitrary implementations. Yep,
you gotta install that code on each slave yourself, but that's no
different from calling out to a ShellCommand to run the python
interpreter out-of-process. Bad idea, good idea?

If someone has an "oh, why don't you just..", that's at least partly
what I'm hoping for.

Thanks

Axel