[Buildbot-devel] Is it possible to deploy buildbot to google app engine

[Marc-Antoine Ruel]

Since I have some GAE experience;

1) Yes, the basic XMPP support could be a workaround to push notifications.
2) No, http-push isn't supported. http hard deadline at 30 seconds.
3) Blobstore[1] could help with regard to log files and other items
like patch files. Billing must be enabled to use this functionality.
4) twisted.internet.defer.Deferred can be more or less converted to
task queue[2] albeit with a 30 seconds limitation.
5) polling is limited to 1 minute[3] latency, except that you can hack
the poller to trigger taskqueue.
6) no svn polling or anything similar. Forget about anything that
calls subprocess.
7) pubsubhubbub[4] can probably be leveraged to workaround some limitations.
8) I did some testing and with the free quota, you'll severely hit the
cpu limit with ~10 queries per seconds (QPS). At 10 QPS, you're at
6.5/24/10 = 27ms/request. Database I/O will usually generate at least
100ms cpu usage per request.
http://chromium-status.appspot.com/current alone receives >12 QPS.
9) We use GAE to push try jobs status inside the codereviews, e.g.
http://codereview.chromium.org/509014 (the green boxes)
10) Dustin is right, the way GAE scales isn't optimal for buildbot
scheduling part. But it's fine for the status serving part so I view
it more like a status proxy than anything else. GAE can be run
standalone for local usage but it won't scale so there isn't much
point for non-public projects. Anyway non-public projects rarely
generate enough status related traffic to kill the master.

So in short, Guo you are asking for a complete rewrite and beside the
status serving part, I don't see much point.

M-A

[1] http://googleappengine.blogspot.com/2009/12/app-engine-sdk-130-released-including.html
[2] http://code.google.com/appengine/docs/python/taskqueue/
[3] http://code.google.com/appengine/docs/python/config/cron.html#The_Schedule_Format
[4] http://code.google.com/p/pubsubhubbub/


On Wed, Dec 30, 2009 at 5:39 AM, Marcus Lindblom <macke at yar.nu> wrote:
> On 2009-12-30 11:27, Dustin Sallings wrote:
>>
>> On Dec 30, 2009, at 2:17, Marcus Lindblom wrote:
>>
>>> I think it'd be awesomely cool to run buildbot instances on GAE, but
>>> that would, as I said, require ripping out quite a bit of the
>>> entrails and re-jigger them into something new, that doesn't store
>>> any state in memory for more than the lifetime of the request.
>>>
>>> The current pickle-system might work on GAE, but we need to
>>> pickle/unpickle a lot (probably too much) on each request. Brian's DB
>>> work would not be applicable, unless it also abstracts the backend
>>> store so much that a non-RDB store is feasible.
>>
>> There's pretty much no part of GAE that could do proper scheduling of
>> builders.  The slaves would all have to be polling it or something --
>> or every slave would have to be internet addressable.  That would be
>> an unfortunate loss of functionality either way.
> Right. Forgot about that direction.
>> XMPP might be a middle man, but XMPP as available to GAE is
>> excessively limited.
> If we don't worry about efficiency too much, does it need to be more
> than sending a notification to the slave that new orders are available
> on the master?
>>
>>>> I made a webhook status notifier and made an excessively simple, but
>>>> event-driven UI here:
>>>>
>>>> http://bbstatus.appspot.com/
>>>>
>>>> source code is here:
>>>>
>>>> http://github.com/dustin/bbstatus
>>>>
>>>> I'd love to see that be touched by someone who knows how to make UIs
>>>> that make sense.  :)
>>>>
>>> Interesting work. Does it poll the build-masters regularly to update
>>> the info?
>>
>> I despise polling.  :)
> As do I. :)
>>
>> It updates as build builders arrive, and as builds and steps start and
>> finish.  The only polling that exists at all is people with web
>> browsers hitting it.
> Ah, so you have a notifer on the buildmaster that pushes changes. That
> makes sense.
>>
>>> Building a web-only-interface for buildbot on GAE should be easy
>>> (if/when we have a HTTP-interface to the DB) and would off-load some
>>> work from the master, so that it could focus on controlling the
>>> slaves and storing results. But, running the entire buildmaster on
>>> GAE would be even cooler. The way GAE scales would make it attractive
>>> to run really large build masters on, handling 100s or thousands of
>>> builds/slaves.
>>
>> GAE scales in a way that is in conflict with things that aren't web
>> services for the most part.
> Well, in my mind BuildBot _is_ mostly a web service (except for the
> continous log output streaming and with some data-push from the server)
> but I might not have the right picture of it.
>
> It seems to me as it's mostly well defined discrete events going back
> and forth. I suppose where it breaks down is where the master notifies
> the slave.
>
> When will a http-push technique be available? It seems as with all the
> AJAX, RSS and similar stuff going on today, there's way too much polling
> going on everywhere.
>
> Cheers,
> /Marcus
>
> ------------------------------------------------------------------------------
> This SF.Net email is sponsored by the Verizon Developer Community
> Take advantage of Verizon's best-in-class app development support
> A streamlined, 14 day to market process makes app distribution fast and easy
> Join now and get one step closer to millions of Verizon customers
> http://p.sf.net/sfu/verizon-dev2dev
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>