[Buildbot-devel] Is it possible to deploy buildbot to google app engine

Marcus Lindblom macke at yar.nu
Wed Dec 30 10:39:45 UTC 2009

On 2009-12-30 11:27, Dustin Sallings wrote:
> On Dec 30, 2009, at 2:17, Marcus Lindblom wrote:
>> I think it'd be awesomely cool to run buildbot instances on GAE, but 
>> that would, as I said, require ripping out quite a bit of the 
>> entrails and re-jigger them into something new, that doesn't store 
>> any state in memory for more than the lifetime of the request.
>> The current pickle-system might work on GAE, but we need to 
>> pickle/unpickle a lot (probably too much) on each request. Brian's DB 
>> work would not be applicable, unless it also abstracts the backend 
>> store so much that a non-RDB store is feasible.
> There's pretty much no part of GAE that could do proper scheduling of 
> builders.  The slaves would all have to be polling it or something -- 
> or every slave would have to be internet addressable.  That would be 
> an unfortunate loss of functionality either way.
Right. Forgot about that direction.
> XMPP might be a middle man, but XMPP as available to GAE is 
> excessively limited.
If we don't worry about efficiency too much, does it need to be more 
than sending a notification to the slave that new orders are available 
on the master?
>>> I made a webhook status notifier and made an excessively simple, but 
>>> event-driven UI here:
>>> http://bbstatus.appspot.com/
>>> source code is here:
>>> http://github.com/dustin/bbstatus
>>> I'd love to see that be touched by someone who knows how to make UIs 
>>> that make sense.  :)
>> Interesting work. Does it poll the build-masters regularly to update 
>> the info?
> I despise polling.  :)
As do I. :)
> It updates as build builders arrive, and as builds and steps start and 
> finish.  The only polling that exists at all is people with web 
> browsers hitting it.
Ah, so you have a notifer on the buildmaster that pushes changes. That 
makes sense.
>> Building a web-only-interface for buildbot on GAE should be easy 
>> (if/when we have a HTTP-interface to the DB) and would off-load some 
>> work from the master, so that it could focus on controlling the 
>> slaves and storing results. But, running the entire buildmaster on 
>> GAE would be even cooler. The way GAE scales would make it attractive 
>> to run really large build masters on, handling 100s or thousands of 
>> builds/slaves.
> GAE scales in a way that is in conflict with things that aren't web 
> services for the most part.
Well, in my mind BuildBot _is_ mostly a web service (except for the 
continous log output streaming and with some data-push from the server) 
but I might not have the right picture of it.

It seems to me as it's mostly well defined discrete events going back 
and forth. I suppose where it breaks down is where the master notifies 
the slave.

When will a http-push technique be available? It seems as with all the 
AJAX, RSS and similar stuff going on today, there's way too much polling 
going on everywhere.


More information about the devel mailing list