[Buildbot-devel] Links to Mozilla's code
Justin Wood (Callek)
callek at gmail.com
Mon Mar 18 01:37:18 UTC 2013
Dustin J. Mitchell wrote:
> A few folks asked for links to Mozilla's Buildbot configs. We're an
> open-source company, so it's all available.. but finding the needles
> in the haystack can be tricky.
> This is Mozilla's fork of Buildbot-0.8.2. The changes are fairly minimal.
> This collects custom implementations of steps, schedulers, factories, etc.
> The Buildbot configuration itself. How all of this comes together is
> a bit over my head.
> I hope that helps someone out. We're at the sprints now, and will be
> for the duration, if you have more questions!
(for those who don't know me, I am "Callek"/Justin Wood from Mozilla Releng)
So to elaborate on this all for everyone here....
What dustin said is accurate though we also use a pre version of
buildbot for most of our slaves, which in practice doesn't change much
since every master is still on 0.8.2 (that 0.8.4-pre is in a branch of
that buildbot repo)
We also setup buildbot in a python virtualenv, with an added
site-package pointing at our tools repo helpers:
Now for the fit-together sense...
We have a _LOT_ of machines, so no single master does all our work.
There are two primary directories we use from buildbot-configs (this
repo is also shared with projects like SeaMonkey):
There are 3 basic types of masters we use:
Scheduler masters are responsible for reading the changesources and
identifying what builds/tests should be scheduled. We drive those with
We can "setup" a master quite easily with our Makefile.setup, which
calls out to a script which knows what to symlink (e.g.
production_config.py to config.py)
All our masters are enumerated in
which the buildbot-configs reads to know if it should omit certain
platforms/etc. (so we don't bloat the config of a master with slaves
that will never connect)
We then take files like
and build deeply nested var configs that help us dictate what
factories/settings should be present for our vast array of build
the bulk of the work happens in a file like
which calls into our buildbotcustom file "misc.py" calling things like
generateBranchObjects which takes a config for a particular branch
(tree) and then creates all the objects that are meant to be used on
Whether that be a windows master, linux master, etc.
There are some known issues in this setup we wish to improve upon, but
this is a setup that at least is working for us right now, at the scale
I hope this summary is both readable/undstandable. But I'm happy to help
clarify any Q's. Either here or in IRC (I'm frequently on
irc.mozilla.org/#releng but sometimes on freenode/#buildbot)
~Justin Wood (Callek)
More information about the devel