[Buildbot-devel] approaching 0.6.0

[Brian Warner]

Just a quick update on the hacking activity of the last few weeks:


Feature: -D TIMESTAMP / -rREVISION
Description: checkouts are performed using a timestamp or revision stamp
Status: working, in use by Twisted (SVN) and my personal buildbot (CVS)

 The timestamp/revisionstamp is used to check out a particular revision of
 the source tree, rather than simply saying "give me the latest". The stamp
 is calculated by a per-VC-system function based upon the set of Changes that
 are going into that particular build. For SVN, it uses the largest revision
 number. For CVS, it uses a timestamp from the middle of the "tree-stable
 timer" region, which should insure that the checkout does not pick up a
 mismatched set of changes.

 This feature still needs testing in a buildbot that uses a BonsaiMailParser,
 since that's the only CVS ChangeSource we have which provides accurate
 timestamps (for everything else we just use the time of receipt of the email
 or CVSToys message).

 Revision stamps are not yet implemented for Darcs or Arch, mostly because I
 don't understand them well enough to do it yet.


Feature: email.MailNotification
Description: sends mail to interested/guilty parties for certain builds
Status: working, in testing on Twisted buildmaster. Some API tweaks remain.

 The MailNotification status target can send mail to the users involved in a
 particular build, or a fixed list of addresses, or both. It can be
 configured to send mail for all builds, only failing builds, or only when a
 build fails when the previous build succeeded. It can be told to include the
 full build logs as attachments to the message.

 Some API improvements are still on my TODO list (like being able to specify
 exactly which build logs are included, since cvs checkouts aren't very
 interesting). Please take a look at buildbot/status/mail.py and tell me what
 you think of the configuration options available.


Feature: debian packaging
Description: buildbot.deb includes init.d startup scripts, /usr/bin/buildbot
Status: somewhat working, needs testing

 Thanks to work by Kirill Lapshin, we should have an installable debian
 package for 0.6.0 . I've added a /usr/bin/buildbot front-end program, which
 can be used to create a new buildmaster or buildslave, and to start/stop and
 existing one. The directories created by this tool can be put in your
 /etc/default/buildbot and an /etc/init.d/ script will start/stop them every
 time your machine boots or shuts down.

 The create-master/start-master aspects of the 'buildbot' tool appear to
 work. The init.d stuff is still untested, and needs some scrutiny to make
 sure the buildbot is run under the correct user (and isn't given root
 privileges, for example). The /usr/bin/buildbot tool still needs
 documentation.

 RPMs are also in the works, thanks to <symbiont> from #twisted.


Other work:

 Fred Drake is setting up the buildbot-commits mailing list. All CVS checkins
 to the buildbot source code will be posted to this read-only list.
 Eventually the meta-buildbot (the buildbot that makes sure the buildbot
 passes its own test suite) will feed off this list.

 Fred is also looking into making the "Force Build" button (on the HTML
 status page) optional, by changing an argument in the config file. I'll be
 making some changes to re-enable force-build (which has been broken since
 the big status rewrite from last month).

 Jonathan Simms (<slyphon> on #twisted) is working on a BuildStep that
 integrates closely with his rewrite of the Twisted unit test framework (the
 original is named Trial, the rewrite is named 'retrial' because 'appeal' was
 apparently already claimed. Twisted: More Bad Puns Per MB Than The
 Competition). This step uses twisted's native RPC system (PB) to report
 detailed test results on a per-test-method basis. This will allow the
 buildmaster to report, e.g., a compact list of just the test cases which
 failed, any log messages that were emitted during the execution of each
 test, and/or the complete stack trace of the exception which caused the test
 to fail.

  This kind of reporting will form the basis of the "Problem" tracking that
  is one of the two biggest post-0.6.0 (and pre-1.0) features I have planned.
  (the other is the "try" build, for which most of the support is already in
  place). Other test-running BuildSteps can get in on the fun if they can
  somehow parse their test output to come up with a list of test cases and
  some separate results for each.

 We're looking into converting the documentation into .texinfo format. The
 distribution tarballs will contain .html and .info derived from these files.

 The metabuildbot is mostly together (running on one of my home machines),
 but I have to trigger it manually, and the status isn't visible anywhere
 outside my home network. I'm working on fixing both of those problems.

 There are some other new features, see the NEWS file for details.


So I think I'm nearly done with features for 0.6.0 . The pieces that I think
need to be added before we can release are:

 force-build needs to be working and configurable

 slave connect/disconnect events need to be displayed on the waterfall page

 the IRC bot should learn some new commands (like "watch build", which will
 watch the current build and make an announcement when it completes)

However, I'm willing to release without these if it looks like they're going
to take a while.

Beyond that, all that remains now is small useability tweaks and bug fixes,
and maybe some compatibility glue. I encourage everyone to grab the code from
CVS and try it out, and report any bugs you find here or in the tracker on
sf.net.

For now, test it on a copy of your buildbot installation rather than
replacing your production server. It is unlikely that you'll be able to use
your old .tap files, as too much internal structure has changed. Sorry. You
will almost certainly need to edit your configuration files to match the new
keywords, but the file format is now more or less documented in the docs/
directory. Many of the Build Factory classes have been refactored mercilessly
(they all take a separate Source-checkout step, for example), and I think
they're much clearer now, but if you use BasicBuildFactory/etc then you'll
need to look at buildbot/process/factory.py and see how you can use the new
classes to accomplish your goals.

Backwards compatiblity: New masters should be able to handle old slaves,
albeit perhaps with some degraded functionality (if they don't, please file a
bug). Specifically, source checkouts were cleaned up considerably, which
changed the master-slave protocol in an incompatible way. Old slaves are only
able to do the update-in-place-to-HEAD action (not -D TIMESTAMP, and not
update-separate-tree-then-copy). The buildslaves now report the version of
their various commands to the master, so it can adapt to old slaves to a
limited extent.

New slaves will probably get confused when talking to an old master.

My goal is to release 0.6.0 by the end of the month. You can help by
hammering on it and reporting bugs. Release early, release often!

cheers,
 -Brian