[Buildbot-devel] Buildbot Summit - Out of process
Dustin J. Mitchell
dustin at v.igoro.us
Wed Nov 24 15:26:43 UTC 2010
On Wed, Nov 24, 2010 at 3:05 AM, Marcus Lindblom <macke at yar.nu> wrote:
> Or am I missing some important point here?
The key point that you didn't mention is that the in-Python status API
is currently synchronous, and that it will have to become asynchronous
when it's backed by a DB.
So every foo.getBar() call will return a deferred. This isn't so bad
for, say, /builders/mybuilder or the like, but it represents a
complete rewrite of the more complex synthesis displays - (t)grid,
waterfall, and console. The JSON calls, in the final analysis, will
basically map an HTTP request to a DB query, and use the usual
Deferred chaining to translate the query results back into JSON. So
by pushing this stuff out of process and onto the client, we get a few
wins:
- no long synchronous rendering procedures on the master
- no need for a second master-side process
- a distinct (and distinctly licensable) web-status project that
others can hack
- only one rewrite :)
Even if we moved the WebStatus to another Python process, without a
significant amount of work on the synthesis displays, it would still
not work - out of process, all of those getBar invocations would be PB
calls, and thus return a Deferred. If you've ever looked at the
waterfall code, this should make you break out in a cold sweat.
By the way, within the buildmaster, I intend to support the status API
in a few languages. That is, you can make the same call via JSON,
from out-of-process Python via PB, or from in-process Python via
regular asynchronous function calls (modulo language differences, of
course). This will give a nice separation between status consumers
and the buildmaster, allowing us to write e.g., status listeners that
can run in-process or out-of-process.
Also, creating a well-defined API gives users a clear indication of
what we are committing to support in future versions. We'll need more
APIs than this one (in particular, what the heck can a BuildStep
subclass do? What *can't* it do?), but this is a big step.
I'm still very open to suggestions here, especially if they come with
an indication that someone other than me will do the work! :)
Dustin
More information about the devel
mailing list