[Buildbot-devel] Request for help for GSoC 2014

[Dustin J. Mitchell]

On Sat, Mar 15, 2014 at 10:51 AM, Damon Wang <damon.devops at gmail.com> wrote:
> As a OpenStack contributor, I pretty know develop both MQ and MQ client are
> big challenges, OpenStack even use a independent project Oslo.message to do
> this job, they use Kombu to deal with RabbitMQ and own code to other MQ. But
> still now I don't think OpenStack's MQ is high-available. As Dustin said,
> use Kombu maybe the best way, Kombu provide almost all back-ends
> (http://kombu.readthedocs.org/en/latest/introduction.html#transport-comparison)
> and lots of API, so we can force on our main problem instead of solving the
> message (and emails, IRC message we pushed) problem all day.

Welcome to Buildbot!  Coincidentally, I've been working with OpenStack
over the last few weeks and have had a look at its MQ implementation,
among other things.  OpenStack's messaging backend *can* be made
highly available, if you're using RabbitMQ with the right
configuration, but that's actually fairly difficult to do.  Still, I
think they've made the right decision to rely on another project for
their MQ needs.  That allows OpenStack components to focus on their
problem, as you say, rather than getting distracted implementing
highly available messaging.

> * Here are some of questions:
>
> I want to know some details about the MQ, I have to admit I'm not very
> familiar with Buildbot, I used one afternoon to scan Buildbot's code, and
> find the MQ related code are almost locate on persistent_queue.py or some
> other files in status, am I right? And should we make new message queue
> compatible with the current one? Or make all codes use the new one? Should
> we package it again and make it compatible with current IQueue? Besides, is
> there more develop documents about Buildbot and its message work principles?

Buildbot has two development branches right now -- master (for the
next release in the 0.8.x series) and nine (for the 0.9.0 release).
The nine branch is where all of the interesting work, including MQ, is
going on.  I'm impressed that you found persistent_queue.py, but
that's actually completely unrelated to the MQ implementation!

Here's the main documentation for the MQ layer (note the "/nine/" in
the URL) - http://docs.buildbot.net/nine/developer/mq.html

I'm sorry you were sent on a false lead -- hopefully this gets you on
the right track?

Dustin