[Buildbot-devel] Add "latent slave" capability to buildbot (e.g., on-demand ec2 slaves)

[Gary Poster]

We (Canonical/Launchpad) are interested in adding what I'm calling
"latent slaves" to buildbot--a class of slaves that can start up
external machines like Amazon EC2 boxes when a test needs to be run.

I'm about to start hacking on this.  My plan is going to need some
changes to buildbot itself.

If anyone is willing to review the plan, I'd appreciate it.  I'd like
some or all of the changes to go into buildbot as a patch eventually, so
I'm interested in approaching this in a way that's palatable to the core
devs.

Primary Goal:

A "latent" EC2 buildbot slave can be added as a slave to a buildbot
configuration.  It is configured with necessary authentication
information to start up EC2 instances, and an AMI to specify the desired
EC2 image (AMI) to use.  When the latent slave gets a build request, it
starts up an EC2 instance of the specified EC2 machine image to run the
build.  At build completion, the EC2 instance shuts down.

Subsidiary Goals:

- EC2 slave instances can hang around for a configurable amount of time
after a build, to possibly start a new build instead of immediately
shutting down.

- "Latent slaves" is, or becomes, a general class of slaves, of which
EC2 is one.

Acceptable Restrictions:

- You probably won't be able to (or want to?) use vanilla machine
images.  Requiring custom prepared machine images is OK.

- EC2 implementation will rely on boto (http://code.google.com/p/boto/)

Implementation Notes:

Note that the "[MINIMAL]" flag notes items that are deficient but
sufficient; they are discussed more after this list.

1. Create a LatentBuildSlave, extending the API of
buildbot.buildslave.BuildSlave.  An abstract latent BuildSlave must
generally know how to start up an instance that will connect to the
master at the right uri and port and with the right name and password.
[MINIMAL] The simplest way to accomplish that with ec2 is to have an
image responsible for the entirety of this (that is, the image would
know the URI and authentication to use to connect with the master).

2. Add a subclass of buildbot.process.builder.SlaveBuilder called
LatentSlaveBuilder.  [MINIMAL] It treats a ``ping`` call without status
(that is, status=None) as a call to ask the LatentBuildSlave to start an
instance.  LatentBuildSlave would have a new method for this purpose.

3. Add an ``addLatentSlave`` to buildbot.process.builder.Builder.  This
will create a LatentSlaveBuilder instance.  It will then call
``self.maybeStartBuild``, like
``buildbot.process.builder.Builder.attached`` does.

Discussion:

- (re: 1. [MINIMAL]) The slickest way to set up a real slave would be to
have the LatentBuildSlave install buildbot on the image itself, but that
sets up a lot of expectations that would be difficult to fulfill across
operating systems.  A simpler improvement to the minimal approach is to
have a convention for images to follow--some mechanism to send botname,
password, master ip, and port to an instance, and to ask the image's
buildbot to start up.  A Twisted pb server waiting for data?  Or perhaps
you provide a configuration script to the buildslave?  I think I'll
start with the minimal approach and see what develops.

- (re 2. [MINIMAL]) Relying on the ``ping`` call without status is
obviously flaky.  ``buildbot.process.builder.startBuild`` could send a
more clear "hey, start up" message than this for LatentBuildSlaves.  I
intend to add one of these, but don't have a spelling yet.

- I don't specify here how the EC2 instance is shut down.  I forget if
BuildSlaves are alerted when a build is completed, but I remember that,
between the LatentBuildSlave amd the LatentSlaveBuilder, I can get the
information, and shut down the EC2 instance if needed.  This should work
out.

- I don't know if buildbot is willing to depend on the boto package.  An
optional dependency?  Or do I need to just put the abstract code in
buildbot, with an EC2 implementation of the LatentBuildSlave in contrib?
That's probably what I'll do without further input.

- This is all the planning I intend to do before I start tackling it.
Getting in the thick of things may change my plans significantly, so
I'll leave the rest for branch review.

Hopefully this'll work. ;-) I'm optimistic.

Gary