[Buildbot-devel] Add "latent slave" capability to buildbot (e.g., on-demand ec2 slaves)

[Gary Poster]

On Tue, 2009-01-06 at 11:14 -0500, Dustin J. Mitchell wrote:
> On Tue, Jan 6, 2009 at 10:19 AM, Gary Poster <gary.poster at canonical.com> wrote:
> > I'm about to start hacking on this.  My plan is going to need some
> > changes to buildbot itself.
> 
> Please hack on the one-oh branch.  The slave manager is just in the
> design stages, so we can make this an integral part of the system,
> rather than an add-on.
>   http://github.com/djmitche/buildbot

That would be cool, but we want to use this within a month or so. :-)
This effort is for our internal development process.  Maybe we will be
able to look at one-oh later.

> > A "latent" EC2 buildbot slave can be added as a slave to a buildbot
> > configuration.  It is configured with necessary authentication
> > information to start up EC2 instances, and an AMI to specify the desired
> > EC2 image (AMI) to use.  When the latent slave gets a build request, it
> > starts up an EC2 instance of the specified EC2 machine image to run the
> > build.  At build completion, the EC2 instance shuts down.
> 
> +1.  You may also want to have some kind of support for multiple
> slaves created with the same creds?  So if you get a flood of
> ubuntu/x86 build requests, you don't have to serialize them.

Yes, precisely.  This would probably be better supported by rethinking
architecture, as you propose.  I have a cheap, imperfect-but-sufficient
plan for that now.

> 
> > - EC2 slave instances can hang around for a configurable amount of time
> > after a build, to possibly start a new build instead of immediately
> > shutting down.
> 
> +1
> 
> > - "Latent slaves" is, or becomes, a general class of slaves, of which
> > EC2 is one.
> 
> +1
> 
> > - You probably won't be able to (or want to?) use vanilla machine
> > images.  Requiring custom prepared machine images is OK.
> 
> I think this is unavoidable - the images will have to have buildbot
> installed, at the very least, right?

Yeah, that's what I'm thinking.

> 
> > - EC2 implementation will rely on boto (http://code.google.com/p/boto/)
> 
> Sounds fine, as long as non-EC2 buildbot can be used without it.
> 
> As for the implementation, I'd rather see this fit the one-oh model.

Understood and appreciated.  The timelines don't match up ATM though.

> > you provide a configuration script to the buildslave?  I think I'll
> > start with the minimal approach and see what develops.
> 
> I don't know enough about EC2 to know what information you can feed a
> newly-minted image, but it would seem that, if we can feed it a very
> short config file, a startup daemon should be able to accomplish the
> rest.

Yeah.  I don't think you can, from my previous EC2/boto experience, but
I'll be investigating that further when I get to that part of the
project.  There's plenty I don't know about that stuff.

> 
> > - (re 2. [MINIMAL]) Relying on the ``ping`` call without status is
> > obviously flaky.  ``buildbot.process.builder.startBuild`` could send a
> > more clear "hey, start up" message than this for LatentBuildSlaves.  I
> > intend to add one of these, but don't have a spelling yet.
> 
> I think, in the one-oh context, that this would be a separate class of
> BuildSlave, and that BuildSlave would do the wakeup when it produces
> an ISlaveEnvironment (getSlaveEnvironment), and do the shutdown some
> configurable time after its last ISlaveEnvironment is gc'd.

Mm, that does sound nice. :-/

> 
> > - I don't know if buildbot is willing to depend on the boto package.  An
> > optional dependency?  Or do I need to just put the abstract code in
> > buildbot, with an EC2 implementation of the LatentBuildSlave in contrib?
> > That's probably what I'll do without further input.
> 
> As long as it's optional, I think it's fine.

Cool.

> 
> > - This is all the planning I intend to do before I start tackling it.
> > Getting in the thick of things may change my plans significantly, so
> > I'll leave the rest for branch review.
> 
> I see that you're following me on github.  As you start hacking, keep
> me (if not the whole list) updated.

Cool, will do.

Thank you for your reply.  I hope this work can make it into one-oh as
well.

Gary