[Buildbot-devel] Add "latent slave" capability to buildbot (e.g., on-demand ec2 slaves)

[Dustin J. Mitchell]

On Tue, Jan 6, 2009 at 10:19 AM, Gary Poster <gary.poster at canonical.com> wrote:
> I'm about to start hacking on this.  My plan is going to need some
> changes to buildbot itself.

Please hack on the one-oh branch.  The slave manager is just in the
design stages, so we can make this an integral part of the system,
rather than an add-on.
  http://github.com/djmitche/buildbot

> A "latent" EC2 buildbot slave can be added as a slave to a buildbot
> configuration.  It is configured with necessary authentication
> information to start up EC2 instances, and an AMI to specify the desired
> EC2 image (AMI) to use.  When the latent slave gets a build request, it
> starts up an EC2 instance of the specified EC2 machine image to run the
> build.  At build completion, the EC2 instance shuts down.

+1.  You may also want to have some kind of support for multiple
slaves created with the same creds?  So if you get a flood of
ubuntu/x86 build requests, you don't have to serialize them.

> - EC2 slave instances can hang around for a configurable amount of time
> after a build, to possibly start a new build instead of immediately
> shutting down.

+1

> - "Latent slaves" is, or becomes, a general class of slaves, of which
> EC2 is one.

+1

> - You probably won't be able to (or want to?) use vanilla machine
> images.  Requiring custom prepared machine images is OK.

I think this is unavoidable - the images will have to have buildbot
installed, at the very least, right?

> - EC2 implementation will rely on boto (http://code.google.com/p/boto/)

Sounds fine, as long as non-EC2 buildbot can be used without it.

As for the implementation, I'd rather see this fit the one-oh model.

> you provide a configuration script to the buildslave?  I think I'll
> start with the minimal approach and see what develops.

I don't know enough about EC2 to know what information you can feed a
newly-minted image, but it would seem that, if we can feed it a very
short config file, a startup daemon should be able to accomplish the
rest.

> - (re 2. [MINIMAL]) Relying on the ``ping`` call without status is
> obviously flaky.  ``buildbot.process.builder.startBuild`` could send a
> more clear "hey, start up" message than this for LatentBuildSlaves.  I
> intend to add one of these, but don't have a spelling yet.

I think, in the one-oh context, that this would be a separate class of
BuildSlave, and that BuildSlave would do the wakeup when it produces
an ISlaveEnvironment (getSlaveEnvironment), and do the shutdown some
configurable time after its last ISlaveEnvironment is gc'd.

> - I don't know if buildbot is willing to depend on the boto package.  An
> optional dependency?  Or do I need to just put the abstract code in
> buildbot, with an EC2 implementation of the LatentBuildSlave in contrib?
> That's probably what I'll do without further input.

As long as it's optional, I think it's fine.

> - This is all the planning I intend to do before I start tackling it.
> Getting in the thick of things may change my plans significantly, so
> I'll leave the rest for branch review.

I see that you're following me on github.  As you start hacking, keep
me (if not the whole list) updated.

Dustin

-- 
Storage Software Engineer
http://www.zmanda.com