[Buildbot-commits] [Buildbot] #1788: MQ options

[Buildbot trac]

#1788: MQ options
-------------------------+--------------------
Reporter:  dustin        |       Owner:
    Type:  project-idea  |      Status:  new
Priority:  major         |   Milestone:  0.9.+
 Version:  0.8.3p1       |  Resolution:
Keywords:                |
-------------------------+--------------------
Description changed by dustin:

Old description:

> When Buildbot adopts an MQ framework, there are a number of choices.
> Among them:
>
>  * pure-python, internal implementation (no network, only a single
> master)
>  * pure-python, internal implementation that uses TCP or UDP between
> masters
>  * http://morbidq.com/
>  * http://code.google.com/p/coilmq/
>  * txamqp
>  * zeromq
>
> The first is already implemented in 'nine', but we need more.
>
> The more difficult part of this project is to ensure that messages are
> only delivered after the corresponding database changes are visible.
>
> The issue is a race condition between message passing and database
> replication.  Imagine you have a large Buildbot installation with several
> replicated MySQL servers and a redundant RabbitMQ cluster.  One buildbot
> master writes changes to a build to the database (an UPDATE operation),
> then sends a message describing the change.  On another master, some
> service gets the message and queries a different MySQL server to see the
> build's new status.  If the message arrives before the database
> replication occurs, then this master will see stale data.  That will lead
> to a lot of subtle, rare bugs.
>
> == scope ==
> This project would involve separate tasks:
>  * implement one or more of the above MQ plugins (mostly just coding)
>  * solve the data-ordering problem (requiring some CS theory)

New description:

 When Buildbot adopts an MQ framework, there are a number of choices.
 Among them:

  * pure-python, internal implementation (no network, only a single master)
  * pure-python, internal implementation that uses TCP or UDP between
 masters
  * http://morbidq.com/
  * http://code.google.com/p/coilmq/
  * txamqp
  * zeromq

 The first is already implemented in 'nine', but we need more.

 The more difficult part of this project is to ensure that messages are
 only delivered after the corresponding database changes are visible.

 The issue is a race condition between message passing and database
 replication.  Imagine you have a large Buildbot installation with several
 replicated MySQL servers and a redundant RabbitMQ cluster.  One buildbot
 master writes changes to a build to the database (an UPDATE operation),
 then sends a message describing the change.  On another master, some
 service gets the message and queries a different MySQL server to see the
 build's new status.  If the message arrives before the database
 replication occurs, then this master will see stale data.  That will lead
 to a lot of subtle, rare bugs.

 == scope ==
 This project would involve separate tasks:
  * implement one or more of the above MQ plugins (mostly just coding)
  * solve the data-ordering problem (requiring some CS theory)

 == relevant bugs ==
 [[TicketQuery(order=priority,status!=closed,keywords~=mq,format=table,col=summary|owner)]]

--

-- 
Ticket URL: <http://trac.buildbot.net/ticket/1788#comment:2>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation