[Buildbot-devel] certain "protocols" of mq will cause queue register failed

[Dustin J. Mitchell]

On Mon, Jun 23, 2014 at 2:49 AM, Damon Wang <damon.devops at gmail.com> wrote:
> Hi,
>
> I tested kombu mq plugin these days, and find lots of queues cant be
> registered successfully, sometimes it raise an error like:
>
> "amqp.exceptions.UnexpectedFrame: Received 0x04 while expecting 0xce"
>
> I tried to use an extra thread for every queue's register, but even the code
> work, the rabbitmq wont (sue rabbitmqctl list_queues and find few queues
> registered successfully), after lots of attempts, I find use
> "llibrabbitmq://guest:guest@localhost//" instead of
> "amqp://guest:guest@localhost//" will solve this problem temporarily.

This seems like a Kombu issue.  I assume you've searched for others
with the same issue?  Have you tried contacting the Kombu community?

A http://sscce.org/ might be helpful to help diagnoes.

> P.S @Dustin: We wont delete exchange when buildbot stop? What about queues?
> It seems that exchange can be declared even it already exists, but it will
> cause exceptions when we use register a queue with a name which already
> exists. Though we can add random letters into queue's name, but there will
> be lots of queues on the mq server if the server haven't restart for a
> while!

So let's imagine that we have a Buildbot installation with three
masters.  Things are humming along nicely, and then we decide to stop
master2 to perform an OS upgrade.  In your model, stopping that master
would delete the exchange.  At that moment, master1 is in the middle
of a build and goes to send a message about starting the "compile"
step.  It publishes a message to .. uh oh, the exchange is gone!

An AMQP terms, declaring an existing resource (exchange, queue, etc.)
is always possible -- that's why they call it "declare" and not
"create".

As for queues: lets pretend "persistent" consumers don't exist for a
week or two, since they're confusing the issue.  Every call to
startConsuming() creates a new, uniquely-named, exclusive queue, bound
to the exchange using the appropriate routing key pattern.  Every call
to stopConsuming() deletes the corresponding queue.  Thus a queue
exists exactly as long as there is something consuming from that
queue.  "Lots of queues" isn't a problem - that's what AMQP does best.

We will need to be careful to balance stopConsuming and startConsuming
calls within the Buildbot source code (and tests can help with this).
Using the 'exclusive' flag for the queues means that the AMQP server
will delete the queues automatically when the connection closes, for
example if a buildmaster crashes.

Dustin