[Buildbot-devel] approaching 0.6.0
warner at lothar.com
Fri Sep 24 09:07:15 UTC 2004
Just a quick update on the hacking activity of the last few weeks:
Feature: -D TIMESTAMP / -rREVISION
Description: checkouts are performed using a timestamp or revision stamp
Status: working, in use by Twisted (SVN) and my personal buildbot (CVS)
The timestamp/revisionstamp is used to check out a particular revision of
the source tree, rather than simply saying "give me the latest". The stamp
is calculated by a per-VC-system function based upon the set of Changes that
are going into that particular build. For SVN, it uses the largest revision
number. For CVS, it uses a timestamp from the middle of the "tree-stable
timer" region, which should insure that the checkout does not pick up a
mismatched set of changes.
This feature still needs testing in a buildbot that uses a BonsaiMailParser,
since that's the only CVS ChangeSource we have which provides accurate
timestamps (for everything else we just use the time of receipt of the email
or CVSToys message).
Revision stamps are not yet implemented for Darcs or Arch, mostly because I
don't understand them well enough to do it yet.
Description: sends mail to interested/guilty parties for certain builds
Status: working, in testing on Twisted buildmaster. Some API tweaks remain.
The MailNotification status target can send mail to the users involved in a
particular build, or a fixed list of addresses, or both. It can be
configured to send mail for all builds, only failing builds, or only when a
build fails when the previous build succeeded. It can be told to include the
full build logs as attachments to the message.
Some API improvements are still on my TODO list (like being able to specify
exactly which build logs are included, since cvs checkouts aren't very
interesting). Please take a look at buildbot/status/mail.py and tell me what
you think of the configuration options available.
Feature: debian packaging
Description: buildbot.deb includes init.d startup scripts, /usr/bin/buildbot
Status: somewhat working, needs testing
Thanks to work by Kirill Lapshin, we should have an installable debian
package for 0.6.0 . I've added a /usr/bin/buildbot front-end program, which
can be used to create a new buildmaster or buildslave, and to start/stop and
existing one. The directories created by this tool can be put in your
/etc/default/buildbot and an /etc/init.d/ script will start/stop them every
time your machine boots or shuts down.
The create-master/start-master aspects of the 'buildbot' tool appear to
work. The init.d stuff is still untested, and needs some scrutiny to make
sure the buildbot is run under the correct user (and isn't given root
privileges, for example). The /usr/bin/buildbot tool still needs
RPMs are also in the works, thanks to <symbiont> from #twisted.
Fred Drake is setting up the buildbot-commits mailing list. All CVS checkins
to the buildbot source code will be posted to this read-only list.
Eventually the meta-buildbot (the buildbot that makes sure the buildbot
passes its own test suite) will feed off this list.
Fred is also looking into making the "Force Build" button (on the HTML
status page) optional, by changing an argument in the config file. I'll be
making some changes to re-enable force-build (which has been broken since
the big status rewrite from last month).
Jonathan Simms (<slyphon> on #twisted) is working on a BuildStep that
integrates closely with his rewrite of the Twisted unit test framework (the
original is named Trial, the rewrite is named 'retrial' because 'appeal' was
apparently already claimed. Twisted: More Bad Puns Per MB Than The
Competition). This step uses twisted's native RPC system (PB) to report
detailed test results on a per-test-method basis. This will allow the
buildmaster to report, e.g., a compact list of just the test cases which
failed, any log messages that were emitted during the execution of each
test, and/or the complete stack trace of the exception which caused the test
This kind of reporting will form the basis of the "Problem" tracking that
is one of the two biggest post-0.6.0 (and pre-1.0) features I have planned.
(the other is the "try" build, for which most of the support is already in
place). Other test-running BuildSteps can get in on the fun if they can
somehow parse their test output to come up with a list of test cases and
some separate results for each.
We're looking into converting the documentation into .texinfo format. The
distribution tarballs will contain .html and .info derived from these files.
The metabuildbot is mostly together (running on one of my home machines),
but I have to trigger it manually, and the status isn't visible anywhere
outside my home network. I'm working on fixing both of those problems.
There are some other new features, see the NEWS file for details.
So I think I'm nearly done with features for 0.6.0 . The pieces that I think
need to be added before we can release are:
force-build needs to be working and configurable
slave connect/disconnect events need to be displayed on the waterfall page
the IRC bot should learn some new commands (like "watch build", which will
watch the current build and make an announcement when it completes)
However, I'm willing to release without these if it looks like they're going
to take a while.
Beyond that, all that remains now is small useability tweaks and bug fixes,
and maybe some compatibility glue. I encourage everyone to grab the code from
CVS and try it out, and report any bugs you find here or in the tracker on
For now, test it on a copy of your buildbot installation rather than
replacing your production server. It is unlikely that you'll be able to use
your old .tap files, as too much internal structure has changed. Sorry. You
will almost certainly need to edit your configuration files to match the new
keywords, but the file format is now more or less documented in the docs/
directory. Many of the Build Factory classes have been refactored mercilessly
(they all take a separate Source-checkout step, for example), and I think
they're much clearer now, but if you use BasicBuildFactory/etc then you'll
need to look at buildbot/process/factory.py and see how you can use the new
classes to accomplish your goals.
Backwards compatiblity: New masters should be able to handle old slaves,
albeit perhaps with some degraded functionality (if they don't, please file a
bug). Specifically, source checkouts were cleaned up considerably, which
changed the master-slave protocol in an incompatible way. Old slaves are only
able to do the update-in-place-to-HEAD action (not -D TIMESTAMP, and not
update-separate-tree-then-copy). The buildslaves now report the version of
their various commands to the master, so it can adapt to old slaves to a
New slaves will probably get confused when talking to an old master.
My goal is to release 0.6.0 by the end of the month. You can help by
hammering on it and reporting bugs. Release early, release often!
More information about the devel