[Buildbot-devel] Buildbot with virtual machines
warner-buildbot at lothar.com
Sat Aug 19 19:32:03 UTC 2006
> Brian, are you listening ??
Yup, just really behind. Some quick thoughts:
I think Stefan's load-balancing support is a good idea. I'd like to see the
existing Locks turned into multiple-holder semaphores (although I was
confused by some of the earlier discussion of readers and writers), and I'd
like to see each buildslave have an "availability" status (set by either the
slave or the master), with values like "eager for builds", "reluctantly
willing to do a build", "not taking on new jobs right now", and of course
completely offline. The request to have buildslaves only get used at night
(when their usual human users are away), or only when their load average is
low, could be implemented with this availability status. A big question is
how to let the buildslave admin configure/control this:
a slave-side crontab running "buildbot set-slave-availability BASEDIR
a slave-side web page where the local admin can click "available"?
some code written into the buildslave? into the buildmaster?
I'd like to avoid lots of extra slave-side command-line options, to keep
things simple, but doing load-average-based availability seems to want to
live inside the buildslave and not in an external tool. I'm reluctant to add
a slave.cfg file, but maybe that's the best way to achieve some of the more
complex goals here.
I would encourage everybody to think in terms of keeping your build
intelligence in your Makefiles and not in the buildmaster, for the sake of
your users who will be wondering how to reproduce the results obtained by
the buildbot. Make a 'test-in-vm' command in your Makefile which locates and
configures and launches the VM tool, waits for it to complete, then somehow
examines the results. Everybody can run 'make test-in-vm' just like the
buildbot does. (note that the VM doesn't necessarily have to run on the same
host as the buildslave.. it could ssh out to a separate [bigger] machine
where the VM images are hosted). Use the buildbot's locking mechanism to
keep the number of concurrent VMs down to a reasonable load.
Given that we're keeping buildbot in Darcs, I'm inclined to have everybody
who's working on new features just publish a Darcs tree that includes
everything from the official repository plus their own patches. Publish the
URL where they're hosted, then everybody who's interested in that feature
can just pull from your repo. When you're comfortable with submitting them
upstream, send me the URL, then I can pull them into my repo, review, and
apply them to the official repository (and CVS).
This is how I do development personally: I have a dozen repositories, one
per feature. For example I have a repo named 'svnURL', where I'm
considering renaming step.SVN's baseurl= argument to baseURL= for
consistency with the other arguments, but I'm still trying to figure out
how to cleanly deprecate the old spelling. I have another named 'stepURL',
where I'm trying to add a buildstep addURL() method, so steps can publish
URLs to external HTML things like code-coverage output, but I'm not
completely comfortable with the semantics yet. When I get these features
ready, I'll push them into my main repository. This is how you do
"branches" in Darcs, but the big advantage is that anybody can do it. All
you need is darcs and a ~/public_html directory somewhere.
(hm, obviously I should be publishing these intermediate repositories too,
so other people can see what I'm working on and tell me whether they're
good or bad ideas. I'll see if I can find a good place to host those..
sf.net's web space too small and slow for this purpose, but I'm slowly
looking for a larger box/colo to host some buildbot development stuff).
One issue I'm still trying to figure out is how to merge the (rather
serialized) ChangeLog entries. I think what I want to do is to use the
top-level ChangeLog file for the official repository (i.e., entries are
only added to it when the change is committed to the canonical repo), but
make a repo.ChangeLog file in which incremental notes are added. When I
merge a repo into the official one, I'll move the repo.ChangeLog entries
into the real ChangeLog and delete repo.ChangeLog .
I'd be happy to add links to any repositories that y'all publish from the
buildbot web page. Also, you should add links to them in the RFEs or bug
reports, so people can search on something like "buildbot load balancing",
discover SF#1524611, see the perhaps-long-running branch where the code is
being developed, and pull from that repository. The one trick is that Darcs
repositories, because they're so easy to create, tend to be short-lived, so
we'll need to remember to remove those links once they've fulfilled their
I'm trying to get a new release out this weekend, just one or two more small
features that I want to sneak in first (those stepURLs in particular). But it
seems like each time I sit down to get it done, something else jumps up to
claim my attention.. isn't that always the way? :)
More information about the devel