[Buildbot-devel] Is it possible to deploy buildbot to google app engine

[Marcus Lindblom]

On 2010-01-04 19:15, Marc-Antoine Ruel wrote:
> Since I have some GAE experience;
>
> 1) Yes, the basic XMPP support could be a workaround to push notifications.
> 2) No, http-push isn't supported. http hard deadline at 30 seconds.
> 3) Blobstore[1] could help with regard to log files and other items
> like patch files. Billing must be enabled to use this functionality.
> 4) twisted.internet.defer.Deferred can be more or less converted to
> task queue[2] albeit with a 30 seconds limitation.
> 5) polling is limited to 1 minute[3] latency, except that you can hack
> the poller to trigger taskqueue.
> 6) no svn polling or anything similar. Forget about anything that
> calls subprocess.
> 7) pubsubhubbub[4] can probably be leveraged to workaround some limitations.
> 8) I did some testing and with the free quota, you'll severely hit the
> cpu limit with ~10 queries per seconds (QPS). At 10 QPS, you're at
> 6.5/24/10 = 27ms/request. Database I/O will usually generate at least
> 100ms cpu usage per request.
> http://chromium-status.appspot.com/current alone receives>12 QPS.
> 9) We use GAE to push try jobs status inside the codereviews, e.g.
> http://codereview.chromium.org/509014 (the green boxes)
> 10) Dustin is right, the way GAE scales isn't optimal for buildbot
> scheduling part. But it's fine for the status serving part so I view
> it more like a status proxy than anything else. GAE can be run
> standalone for local usage but it won't scale so there isn't much
> point for non-public projects. Anyway non-public projects rarely
> generate enough status related traffic to kill the master.
>
> So in short, Guo you are asking for a complete rewrite and beside the
> status serving part, I don't see much point.

So, a GAEStatus pushing info and something similar to Dustin's stuff 
running on GAE would be what we'd want, if we go there.
(i.e. building a web-view, probably using the current jinja templates 
but with a different back-end.)

However, I won't try something until BlobStore is available for 
non-billing projects. (Heck, 100 mb max would probably be enough, given 
that old logs aren't _that_ important.)

Thanks for the informative writeup! :)

Cheers,
/Marcus