[Buildbot-devel] Buildbot Summit - Out of process

[Marcus Lindblom]

On 2010-11-24 16:26, Dustin J. Mitchell wrote:
> On Wed, Nov 24, 2010 at 3:05 AM, Marcus Lindblom<macke at yar.nu>  wrote:
>> Or am I missing some important point here?
>
> The key point that you didn't mention is that the in-Python status API
> is currently synchronous, and that it will have to become asynchronous
> when it's backed by a DB.

Ah, so the current JSON is plagued by the same thing, and those details 
will change when we get DB-over-JSON. I got it.

> So every foo.getBar() call will return a deferred.  This isn't so bad
> for, say, /builders/mybuilder or the like, but it represents a
> complete rewrite of the more complex synthesis displays - (t)grid,
> waterfall, and console.  The JSON calls, in the final analysis, will
> basically map an HTTP request to a DB query, and use the usual
> Deferred chaining to translate the query results back into JSON.  So
> by pushing this stuff out of process and onto the client, we get a few
> wins:
>
>   - no long synchronous rendering procedures on the master
>   - no need for a second master-side process
>   - a distinct (and distinctly licensable) web-status project that
> others can hack
>   - only one rewrite :)
>
> Even if we moved the WebStatus to another Python process, without a
> significant amount of work on the synthesis displays, it would still
> not work - out of process, all of those getBar invocations would be PB
> calls, and thus return a Deferred.  If you've ever looked at the
> waterfall code, this should make you break out in a cold sweat.

Righto. So there will be some cold sweat anyhow to make this work in the 
future, and perhaps a JSON "DB-API" is the way forward. I hope it'll be 
able to do some SQL-like aggregated queries (I.e. get a bunch of builds 
or so, not just one at a a time... I'm not talking complex joins or 
selects here..)

> By the way, within the buildmaster, I intend to support the status API
> in a few languages.  That is, you can make the same call via JSON,
> from out-of-process Python via PB, or from in-process Python via
> regular asynchronous function calls (modulo language differences, of
> course).  This will give a nice separation between status consumers
> and the buildmaster, allowing us to write e.g., status listeners that
> can run in-process or out-of-process.

Fair enough.

> Also, creating a well-defined API gives users a clear indication of
> what we are committing to support in future versions.  We'll need more
> APIs than this one (in particular, what the heck can a BuildStep
> subclass do? What *can't* it do?), but this is a big step.
>
> I'm still very open to suggestions here, especially if they come with
> an indication that someone other than me will do the work! :)

Gimme a JSON/DB API and I'll maybe have a go at updating the WebStatus.
It'd be interesting to see what kind of caching is necessary/doable to 
get a separate WebStatus up and running.

Also, I can then write it in Django, just for fun ;-P

Cheers,
/Marcus