[Buildbot-devel] Buildbot Summit - Out of process

[Dustin J. Mitchell]

On Tue, Nov 23, 2010 at 9:46 AM, Marc-Antoine Ruel <maruel at chromium.org> wrote:
> My main point is: have the web/json/mail status serving in a separate
> process but reusing the same twisted infrastructure, so status/ classes can
> be reused as is in a separate project transparently. I'm probably dreaming
> though. It requires having fully bidirectional communication to control
> jobs. I should stop writing emails and go back to coding instead.

That's a lot harder than it sounds, since most of the status calls
that the web status makes are synchronous.  I'd rather separate it out
as we discussed, with client-side assembly of views and JSON requests
for state and events.

> As most Moz folks were advocating for db-wise cross process communication, I
> prefer socket based one since it's easier to deploy the status serving in a
> remote DC but in the end it doesn't matter, whatever works first wins.

We're beginning to regret that decision.  There's some work within
Mozilla to set up Pulse, which is an event broker through which all of
the Mozilla firehose (bugzilla changes, build results, commits, etc.)
will flow.  Since we'll have that expertise in place, I'm thinking of
replacing the clunky and awful RDBMS-as-message-queue with a more
MQ-like system.  This would map nicely to the need to stream events
from multiple events to every event-listening client: basically,
extend the message queue to the event-push system.  This is still a
relatively unformed idea - I'd like to get some experience with the
relevant technologies via Pulse, and then see how to apply that
experience to Buildbot.

Single-master configurations will, of course, still be usable with
minimal additional equipment.

Dustin