[Buildbot-devel] GSoC 2012 proposal for Implementation of a new Master-Slave Protocol.

[Saurabh Kumar]

On Wed, Apr 4, 2012 at 10:56 AM, Tom Prince <tom.prince at ualberta.net> wrote:
> The data should be json *serializable*, but not necessarily be
> serialized that way. but amp for example, has a native serialization of
> all the types that json can serialize, and that should be used when
> using amp.

Yes,  I agree. The backends can each have their serialization format,
using JSON when none is provided.

> I'd be interested in seeing what shape you see the mutlt-master scaling
> taking. As well as the graceful handling of disconnections.

I thought of broker based designs, where the masters connect to one of
several synchronized brokers, and slaves connect to the same brokers.
The broker has knowledge of the masters and slaves present and types
of the slaves, and adds to its knowledge whenever a new master or
slave connects. One master can send out messages to all connected
slaves without having knowledge about the number of them being present
and one slave can also initiate connection with a master that broker
tells the slave about. This is naive, and probably can be improved and
perhaps can be done without brokers.
Graceful disconnection would mean storing messages in a queue, so that
messages to a slave or master aren't lost. If reconnection fails, then
they can be sent to a different slave or master. The broker would know
about the slave or master when it comes back online. Some form of
identification would ensure that its the same slave or master as the
one disconnected before.

> I know it is hard to estimate in advance, particularly when a the design
> isn't fixed yet. But I'd be interested to know if you have any thoughts
> on what kind of time-frame this would take.

I am finding it increasingly difficult to specify this, since I
haven't done a project this big before. A lot of it would depend on
which parts of my application design you agree to. I had thought that
implementing each of them could take two weeks each and then I could
look amqp or celery+rabbitmq or others. Now I think ensuring backward
compatibility would also add about two to three weeks into the mix
leaving very little time for me to add other backends.

--Saurabh