[Buildbot-devel] Buildbot Summit - Out of process

[Dustin J. Mitchell]

On Wed, Nov 24, 2010 at 3:05 AM, Marcus Lindblom <macke at yar.nu> wrote:
> Or am I missing some important point here?

The key point that you didn't mention is that the in-Python status API
is currently synchronous, and that it will have to become asynchronous
when it's backed by a DB.

So every foo.getBar() call will return a deferred.  This isn't so bad
for, say, /builders/mybuilder or the like, but it represents a
complete rewrite of the more complex synthesis displays - (t)grid,
waterfall, and console.  The JSON calls, in the final analysis, will
basically map an HTTP request to a DB query, and use the usual
Deferred chaining to translate the query results back into JSON.  So
by pushing this stuff out of process and onto the client, we get a few
wins:

 - no long synchronous rendering procedures on the master
 - no need for a second master-side process
 - a distinct (and distinctly licensable) web-status project that
others can hack
 - only one rewrite :)

Even if we moved the WebStatus to another Python process, without a
significant amount of work on the synthesis displays, it would still
not work - out of process, all of those getBar invocations would be PB
calls, and thus return a Deferred.  If you've ever looked at the
waterfall code, this should make you break out in a cold sweat.

By the way, within the buildmaster, I intend to support the status API
in a few languages.  That is, you can make the same call via JSON,
from out-of-process Python via PB, or from in-process Python via
regular asynchronous function calls (modulo language differences, of
course).  This will give a nice separation between status consumers
and the buildmaster, allowing us to write e.g., status listeners that
can run in-process or out-of-process.

Also, creating a well-defined API gives users a clear indication of
what we are committing to support in future versions.  We'll need more
APIs than this one (in particular, what the heck can a BuildStep
subclass do? What *can't* it do?), but this is a big step.

I'm still very open to suggestions here, especially if they come with
an indication that someone other than me will do the work! :)

Dustin