[Buildbot-devel] Buildbot with virtual machines

[Brian Warner]

> Brian, are you listening ??

Yup, just really behind. Some quick thoughts:


Load-balancing:

 I think Stefan's load-balancing support is a good idea. I'd like to see the
 existing Locks turned into multiple-holder semaphores (although I was
 confused by some of the earlier discussion of readers and writers), and I'd
 like to see each buildslave have an "availability" status (set by either the
 slave or the master), with values like "eager for builds", "reluctantly
 willing to do a build", "not taking on new jobs right now", and of course
 completely offline. The request to have buildslaves only get used at night
 (when their usual human users are away), or only when their load average is
 low, could be implemented with this availability status. A big question is
 how to let the buildslave admin configure/control this:

  a slave-side crontab running "buildbot set-slave-availability BASEDIR
  unavailable" ?

  a slave-side web page where the local admin can click "available"?

  some code written into the buildslave? into the buildmaster?

 I'd like to avoid lots of extra slave-side command-line options, to keep
 things simple, but doing load-average-based availability seems to want to
 live inside the buildslave and not in an external tool. I'm reluctant to add
 a slave.cfg file, but maybe that's the best way to achieve some of the more
 complex goals here.


VM buildslaves:

 I would encourage everybody to think in terms of keeping your build
 intelligence in your Makefiles and not in the buildmaster, for the sake of
 your users who will be wondering how to reproduce the results obtained by
 the buildbot. Make a 'test-in-vm' command in your Makefile which locates and
 configures and launches the VM tool, waits for it to complete, then somehow
 examines the results. Everybody can run 'make test-in-vm' just like the
 buildbot does. (note that the VM doesn't necessarily have to run on the same
 host as the buildslave.. it could ssh out to a separate [bigger] machine
 where the VM images are hosted). Use the buildbot's locking mechanism to
 keep the number of concurrent VMs down to a reasonable load.


Sandbox branch:

 Given that we're keeping buildbot in Darcs, I'm inclined to have everybody
 who's working on new features just publish a Darcs tree that includes
 everything from the official repository plus their own patches. Publish the
 URL where they're hosted, then everybody who's interested in that feature
 can just pull from your repo. When you're comfortable with submitting them
 upstream, send me the URL, then I can pull them into my repo, review, and
 apply them to the official repository (and CVS).

  This is how I do development personally: I have a dozen repositories, one
  per feature. For example I have a repo named 'svnURL', where I'm
  considering renaming step.SVN's baseurl= argument to baseURL= for
  consistency with the other arguments, but I'm still trying to figure out
  how to cleanly deprecate the old spelling. I have another named 'stepURL',
  where I'm trying to add a buildstep addURL() method, so steps can publish
  URLs to external HTML things like code-coverage output, but I'm not
  completely comfortable with the semantics yet. When I get these features
  ready, I'll push them into my main repository. This is how you do
  "branches" in Darcs, but the big advantage is that anybody can do it. All
  you need is darcs and a ~/public_html directory somewhere.

   (hm, obviously I should be publishing these intermediate repositories too,
   so other people can see what I'm working on and tell me whether they're
   good or bad ideas. I'll see if I can find a good place to host those..
   sf.net's web space too small and slow for this purpose, but I'm slowly
   looking for a larger box/colo to host some buildbot development stuff).

  One issue I'm still trying to figure out is how to merge the (rather
  serialized) ChangeLog entries. I think what I want to do is to use the
  top-level ChangeLog file for the official repository (i.e., entries are
  only added to it when the change is committed to the canonical repo), but
  make a repo.ChangeLog file in which incremental notes are added. When I
  merge a repo into the official one, I'll move the repo.ChangeLog entries
  into the real ChangeLog and delete repo.ChangeLog .

 I'd be happy to add links to any repositories that y'all publish from the
 buildbot web page. Also, you should add links to them in the RFEs or bug
 reports, so people can search on something like "buildbot load balancing",
 discover SF#1524611, see the perhaps-long-running branch where the code is
 being developed, and pull from that repository. The one trick is that Darcs
 repositories, because they're so easy to create, tend to be short-lived, so
 we'll need to remember to remove those links once they've fulfilled their
 purpose.



I'm trying to get a new release out this weekend, just one or two more small
features that I want to sneak in first (those stepURLs in particular). But it
seems like each time I sit down to get it done, something else jumps up to
claim my attention.. isn't that always the way? :)

cheers,
 -Brian