[Buildbot-devel] Latent Slave code changes

[John Carr]

Hi guys,

I'd like to make some changes to latent slaves to better support starting virtual machines on demand. A working prototype of the changes i'd like to make are here, but i'd like to clarify a few things. I dont expect that this is ready to land.

http://github.com/Jc2k/buildbot/commits/master/

The rationale behind this change is that I need to support many different VM configurations, but if i configure them all then there is no way to stop buildbot starting all of them at the same time (builder locks are too late in the process). In particular the fragile old laptop i'm testing on can't really handle running 5 VM's at the same time. So i needed to find the last place in the code before a latent slave actually is started where it was possible to do locking and fail *nicely* if a lock couldnt be acquired.

In the above branch, start_instance now returns True if a latent slave is started or False if it is not. This is chained together so that if a VM couldnt be started a new attempt to find a slave occurs. (It is expected that any implementation returning False will be implementing canStartBuild so this won't cause any looping).

So outstanding issues.

 * I'm new to this code, who knows if what I did is even close :D

 * I'm breaking API, but the only other way involves a cludgey use of an errback when its not really an error. raise ICantStartAThingButDontPanicJustTryLater :)

 * The current code allows start_instance to return things to tag on an event. This is still there, but is no longer getting anything useful from the return value of start_instance. Option 1 is to do the cludge i've already mentioned. Option 2 is living with it. Option 3 is letting the AbstractLatentBuildSlave implementer do this by themselves if they want it. Option 4 is another method on the slave that can return something useful to attach to the event. (See the substantiate method in process/builder.py).

 * How do I test it - i really want to do something like the "runs" tests in broken_tests. Right now i'm testing it by using it but it can make it awkward to reproduce and nail bugs :)

 * If I can get these patches in then it should be possible to build on it and let any old BuildSlave take a list of locks. Would that be useful?

If I can get this in I shouldnt be too far off submitting a libvirtbuildslave.py too...

John