[Buildbot-devel] Is it possible to deploy buildbot to google app engine

[Marcus Lindblom]

On 2009-12-30 11:27, Dustin Sallings wrote:
>
> On Dec 30, 2009, at 2:17, Marcus Lindblom wrote:
>
>> I think it'd be awesomely cool to run buildbot instances on GAE, but 
>> that would, as I said, require ripping out quite a bit of the 
>> entrails and re-jigger them into something new, that doesn't store 
>> any state in memory for more than the lifetime of the request.
>>
>> The current pickle-system might work on GAE, but we need to 
>> pickle/unpickle a lot (probably too much) on each request. Brian's DB 
>> work would not be applicable, unless it also abstracts the backend 
>> store so much that a non-RDB store is feasible.
>
> There's pretty much no part of GAE that could do proper scheduling of 
> builders.  The slaves would all have to be polling it or something -- 
> or every slave would have to be internet addressable.  That would be 
> an unfortunate loss of functionality either way.
Right. Forgot about that direction.
> XMPP might be a middle man, but XMPP as available to GAE is 
> excessively limited.
If we don't worry about efficiency too much, does it need to be more 
than sending a notification to the slave that new orders are available 
on the master?
>
>>> I made a webhook status notifier and made an excessively simple, but 
>>> event-driven UI here:
>>>
>>> http://bbstatus.appspot.com/
>>>
>>> source code is here:
>>>
>>> http://github.com/dustin/bbstatus
>>>
>>> I'd love to see that be touched by someone who knows how to make UIs 
>>> that make sense.  :)
>>>
>> Interesting work. Does it poll the build-masters regularly to update 
>> the info?
>
> I despise polling.  :)
As do I. :)
>
> It updates as build builders arrive, and as builds and steps start and 
> finish.  The only polling that exists at all is people with web 
> browsers hitting it.
Ah, so you have a notifer on the buildmaster that pushes changes. That 
makes sense.
>
>> Building a web-only-interface for buildbot on GAE should be easy 
>> (if/when we have a HTTP-interface to the DB) and would off-load some 
>> work from the master, so that it could focus on controlling the 
>> slaves and storing results. But, running the entire buildmaster on 
>> GAE would be even cooler. The way GAE scales would make it attractive 
>> to run really large build masters on, handling 100s or thousands of 
>> builds/slaves.
>
> GAE scales in a way that is in conflict with things that aren't web 
> services for the most part.
Well, in my mind BuildBot _is_ mostly a web service (except for the 
continous log output streaming and with some data-push from the server) 
but I might not have the right picture of it.

It seems to me as it's mostly well defined discrete events going back 
and forth. I suppose where it breaks down is where the master notifies 
the slave.

When will a http-push technique be available? It seems as with all the 
AJAX, RSS and similar stuff going on today, there's way too much polling 
going on everywhere.

Cheers,
/Marcus