[Buildbot-devel] MQ design plan and code review

[Damon Wang]

Hi all,

I've tried implement our new mq for weeks, after get more and more
information of kombu, I have some ideas to share and some questions to ask.

First, I have some a news, that kombu doesn't support Qpid as we expect, if
we want to keep our doc (http://docs.buildbot.net/nine/developer/mq.html)
's accuracy, we must implement another mq plugin by qpid-python.

Besides, we use persistent to make sure the message will send to a consumer
as soon as it active, but if we use mq, the message will always available
in the mq, so persistent can be ignored here, but our new mq has another
attr -- durable, durable queues remain active when a server restarts, my
design is in default, exchange and queues will establish when buildbot
start, and delete when buildbot shutdown, durable can help us when buildbot
stop anomaly, so queues are not deleted as expect, then we can get our
messages which not consumed.


What's more, our former ued filter to make consumer get right message,
standard mq often make consumer bind to a explicit queue, and will get all
messages from this queue, do we still need filter, this can implement by
filter message's routing_key after our consumer get it from queue, but I
don't know whether it is necessary.

About serializer, we use json as doc says, but does json support datetime
type in python? I find some message contain a time info like this:
"'complete_at': datetime.datetime(2014, 6, 14, 1, 47, 39,
tzinfo=<buildbot.util.UTC object at 0x2c8b7d0>)", but when I test in my
environment, I got "datetime.datetime(2014, 6, 14, 1, 47, 39) is not JSON
serializable", does buildbot has extend json?


Last but not least, to consume message asynchronous, our former mq use
twisted's derfer, I'm not very familiar with it, but after read twisted's
doc, I find it is not easy to use defer here, first, kombu consume messages
by something like: "connection.drain_events", this function will return
when any consumer get a message (event), or use hub(kombu's asyn method),
all of this are face to connection not certain queue or consumer, so make a
thread to run hub maybe a solution, but I know that twisted's aysn IO is
very famous, there must be a solution better than use system's thread or
process, please make comments if you have any.

The current code is on my github, link is
https://github.com/MatheMatrix/Gist/blob/master/kumbu.test.buildbot.2.py,
maybe I can make a pull request for review? Not for merge, only for
convenience to comment.

Thanks,
Damon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20140614/3a68faf7/attachment.html>