[Buildbot-devel] MQ design plan and code review

[Dustin J. Mitchell]

On Wed, Jun 18, 2014 at 5:55 AM, Damon Wang <damon.devops at gmail.com> wrote:
> I have a question that all master can connect one amqp server, so masters will see all queues?

Yes, well, one cluster of amqp servers.

> Besides, I wanna share my new design plan, set all queues at start is a bad idea indeed, especially if we set lots of queues, I do this for setup consumers easily -- we can give consumer the certain queue, but how to set consumer when we do not set queues at start? My idea is to search the queue list first and establish one if doesn't exist, we can make the queue list a dict, and use routing_key as dict's keys, so we just get queues["builder.*.started"], and catch exception.

When a (non-persistent) consumer starts, declare a queue for that
consumer with exclusive=True and some automatically generated unique
name, and then bind that queue to the exchange using the supplied
routing key patterns.  When that consumer stops, delete the queue.

Your suggestion would have two problems:
 1. Since queues are never deleted, queues which no longer have an
attached consumer would fill up and cause the amqp server to fail
 2. Two startConsuming calls with the same routing key pattern would
consume from the same queue, so any message in that queue would only
go to one of the consumers -- not what we want.

Dustin