[Buildbot-devel] the aftermath (was Re: SF bay area get-together?)

[Dustin J. Mitchell]

On Wed, Aug 6, 2008 at 9:55 AM, Axel Hecht <l10n.moz at googlemail.com> wrote:
> I'm not so worried about buildmaster write performance, i.e., having
> db writes being blocking. Most masters don't write that often, and in
> those setups where they do, things like having the db master locally
> and let the web interface work on remote slave dbs or something sounds
> like the more standard way to attack that problem then to start doing
> bookkeeping asynch.

This sync/async quesiton is *important*, but let's be clear about how
much it would cost.  Currently, all of our "DB" operations (pickling,
unpickling) are blocking.  Furthermore, our web-page creation is also
blocking (the grid and waterfall can both take several seconds to
render, during which time no interaction occurs with buildslaves, nor
with other web clients).  I would very much like to get *away* from
that particular mode of operation, but I think that the performance
gains of a database over pickles would be significant even if the DB
API was blocking.

> One thing that I found really hard to think about was trying to port
> the feature of incremental log display into a remote web front end. I
> figured one could have a specific back-door for this in the master,
> but I have no idea how to get the callback oriented data delivery
> ported over into something like django, which assumes to get the data
> all at once.

I've been assuming that logfiles will be stored on-disk as distinct
files, even with a DB.  My config.log for every build is 650K, which
is a mighty big BLOB.

Dustin

-- 
Storage Software Engineer
http://www.zmanda.com