[Buildbot-devel] HTML user manuals now online

[Brian Warner]

I've HTMLified the .texinfo manuals from both 0.6.6 and current CVS, and put
links to them on the main http://buildbot.sf.net web page, along with a
gentle warning about the perils of each. Please take a look and send any
changes my way (preferably as patches to the .texinfo source, but I'll take
anything). Thanks to everybody for the encouragement: I've finally come to
agree that incomplete and/or incorrect is better than non-existent.

I've converted my internal metabuildbot (the one that runs the buildbot unit
tests) over to use Schedulers, and survived to tell the tale. There were a
number of bugs not caught by the unit tests which made a difference, for
which the best solution was to add even more unit tests. I have a Lock
working as advertised (at least it's doing so after I updated the manual to
advertise the behavior that I actually implemented), and a single one-branch
Scheduler.

The biggest problem right now is the lack of a way to reveal what happens
between the time a Change is received and the time a Build starts. The old
scheme had each Change get distributed to all Builders at the same time, so
it made sense to say that a Builder was in the "waiting" state. But in the
new scheme, it is the Schedulers which are in the "waiting" state, and there
is not yet a place on the Waterfall page to show Scheduler status, nor is
there an ISchedulerStatus interface defined yet. And really it's a bit more
complex than that, each Scheduler can have a queue of BuildSets waiting for
the tree to settle down, each Scheduler can feed an arbitrary (and possibly
overlapping) set of Builders, and each Builder can have a queue of Builds
waiting for a slave (possibly one of many) to become available. There is a
lot more information to display than we had before.

I'm thinking that for 0.7.0 I will create ISchedulerStatus and let you get at
the list of current Schedulers (and watch for events, etc, just like with
Builders), but I won't yet provide any sort of historical information. I
think this will be enough to add a row onto the Waterfall that shows which
Schedulers are in which states. But it won't let you ask questions like "what
happened to Change #39", where the idea answer would be a list of Schedulers
who paid attention to it, a list of Builds that used it, and a list of
BuildSets that either passed or failed because of it. After 0.7.0 we can look
at this again and figure out what kind of interface would be most useful.

I think that I'll need to create a couple of new HTML pages to represent
everything, some chronological (like the Waterfall page, but with different
columns), and some that only show the latest event. There will probably be a
page that has one column per Scheduler, with a list of "waiting" Changes, and
a list of not-yet-finished BuildSets (and maybe the most-recently-completed
BuildSet in a third row). There will probably also be a page that just shows
what each slave is doing right now, so one column per buildslave, one row
that shows the current Build being run by that slave (or empty), and one row
with information about the slave. Really I'd like to move to Nevow for all
this web stuff, but it isn't going to happen for 0.7.0: too much work, too
many new things for me to learn, and too large of a dependency to add right
now. But it will happen at some point, and when that does happen, buildbot
will probably require twisted-2.0.x (since the next version of Nevow will
require it). There are plenty of good reasons to upgrade to the current
Twisted release.. when you see the opportunity, take it.

So, I will cautiously say that the code in CVS is functional, at least if
you're willing to update your config files and deal with the holes in the
status output. I may want to change around the way that the new features are
configured before 0.7.0 is released: I keep going back and forth about
whether it is best to put strings or actual objects in master.cfg . I've
already hit one bug that resulted from my use of objects there (if you use
Locks in current CVS, I recommend stopping/restarting the buildmaster when
modifying the Locks in the config file). I'll probably have a better feel for
where the syntax is going when I convert the Twisted buildmaster over to the
new code, probably in about two weeks. (I will be mostly offline next week,
so I'm not going to try to make any sudden changes until I'll be available to
fix the damage).

enjoy!,
 -Brian