[Buildbot-devel] Request for help for GSoC 2014

[Damon Wang]

Hi,

Thanks for you all replies.

@Dustin:

I made a mistake that I found related code in BuildBot's master branch, I
should checkout to nine :-)

I read BuildBot's code these days, find the current mq is very small but it
works, though some test code not done. BuildBot's remote things (message,
keep-alive, or other things) rely on Perspective Broker, Message Queue and
Database, right? I'm not understand why keep-alive use Prospective Broker
instead of mq, seems a historical problem? MQ now seems used widely, search
'self.master.mq' will get lots of results :-)

Here are some of my observations, I' not sure it is right or not:

BuildMaster.create_child_services create mq using mqconnector.MQConnector
and then this mq will be used as self.master.mq in many cases.

A consuming example: BuildMaster.create_child_services create botmaster(a
object of BotMaster), and its startService use mq.startConsuming to consume
a message, and once this messgae consumed, the callback
function(buildRequestAdded) will run. And my question: the startConsuming
will always wait until one message match the filter come?

A produce example: BotMaster.__init__ create brd(an object of
BuildRequestDistributor), and brd._maybeStartBuildsOnBuilder (this function
will go away?) will publish a message by self.master.mq.produce once claim
successfully.

I don't think I'm familiar with BuildBot's code, so if my understand is
wrong, please do not hesitate to point out.

@Pierre:

Thanks for your information, I'll try my best to solve them :-)

Many thanks,
Damon


2014-03-19 4:05 GMT+08:00 Pierre Tardy <tardyp at gmail.com>:

> Also, you will find part of our plans on mq in the trac db:
>
> http://trac.buildbot.net/query?status=!closed&keywords=~mq
>
> especially this one:
> http://trac.buildbot.net/ticket/2644
>
>
> Pierre
>
>
> On Tue, Mar 18, 2014 at 2:47 PM, Dustin J. Mitchell <dustin at v.igoro.us>wrote:
>
>> On Sat, Mar 15, 2014 at 10:51 AM, Damon Wang <damon.devops at gmail.com>
>> wrote:
>> > As a OpenStack contributor, I pretty know develop both MQ and MQ client
>> are
>> > big challenges, OpenStack even use a independent project Oslo.message
>> to do
>> > this job, they use Kombu to deal with RabbitMQ and own code to other
>> MQ. But
>> > still now I don't think OpenStack's MQ is high-available. As Dustin
>> said,
>> > use Kombu maybe the best way, Kombu provide almost all back-ends
>> > (
>> http://kombu.readthedocs.org/en/latest/introduction.html#transport-comparison
>> )
>> > and lots of API, so we can force on our main problem instead of solving
>> the
>> > message (and emails, IRC message we pushed) problem all day.
>>
>> Welcome to Buildbot!  Coincidentally, I've been working with OpenStack
>> over the last few weeks and have had a look at its MQ implementation,
>> among other things.  OpenStack's messaging backend *can* be made
>> highly available, if you're using RabbitMQ with the right
>> configuration, but that's actually fairly difficult to do.  Still, I
>> think they've made the right decision to rely on another project for
>> their MQ needs.  That allows OpenStack components to focus on their
>> problem, as you say, rather than getting distracted implementing
>> highly available messaging.
>>
>> > * Here are some of questions:
>> >
>> > I want to know some details about the MQ, I have to admit I'm not very
>> > familiar with Buildbot, I used one afternoon to scan Buildbot's code,
>> and
>> > find the MQ related code are almost locate on persistent_queue.py or
>> some
>> > other files in status, am I right? And should we make new message queue
>> > compatible with the current one? Or make all codes use the new one?
>> Should
>> > we package it again and make it compatible with current IQueue?
>> Besides, is
>> > there more develop documents about Buildbot and its message work
>> principles?
>>
>> Buildbot has two development branches right now -- master (for the
>> next release in the 0.8.x series) and nine (for the 0.9.0 release).
>> The nine branch is where all of the interesting work, including MQ, is
>> going on.  I'm impressed that you found persistent_queue.py, but
>> that's actually completely unrelated to the MQ implementation!
>>
>> Here's the main documentation for the MQ layer (note the "/nine/" in
>> the URL) - http://docs.buildbot.net/nine/developer/mq.html
>>
>> I'm sorry you were sent on a false lead -- hopefully this gets you on
>> the right track?
>>
>> Dustin
>>
>>
>> ------------------------------------------------------------------------------
>> Learn Graph Databases - Download FREE O'Reilly Book
>> "Graph Databases" is the definitive new guide to graph databases and their
>> applications. Written by three acclaimed leaders in the field,
>> this first edition is now available. Download your free book today!
>> http://p.sf.net/sfu/13534_NeoTech
>> _______________________________________________
>> Buildbot-devel mailing list
>> Buildbot-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20140321/98f8f38d/attachment.html>