[Buildbot-devel] GSoC 2012 proposal for Implementation of a new Master-Slave Protocol.

[Tom Prince]

On Wed, 4 Apr 2012 21:38:30 +0530, Saurabh Kumar <saurabhgeek92 at gmail.com> wrote:
> > I'd be interested in seeing what shape you see the mutlt-master scaling
> > taking. As well as the graceful handling of disconnections.
> 
> I thought of broker based designs, where the masters connect to one of
> several synchronized brokers, and slaves connect to the same brokers.
> The broker has knowledge of the masters and slaves present and types
> of the slaves, and adds to its knowledge whenever a new master or
> slave connects. One master can send out messages to all connected
> slaves without having knowledge about the number of them being present
> and one slave can also initiate connection with a master that broker
> tells the slave about. This is naive, and probably can be improved and
> perhaps can be done without brokers.
> Graceful disconnection would mean storing messages in a queue, so that
> messages to a slave or master aren't lost. If reconnection fails, then
> they can be sent to a different slave or master. The broker would know
> about the slave or master when it comes back online. Some form of
> identification would ensure that its the same slave or master as the
> one disconnected before.

This looks interesting and useful. Doing something like this changes how
the master and slave interact in a fairly fundamental way. This isn't a
bad thing (at all), but does have an effect on the scope of necessary
changes. As djmitche suggested, it may or may not be feasible to
implement this all at once, but it will certainly be worthwhile to at
least plan how things like this can be integrated.

> > I know it is hard to estimate in advance, particularly when a the design
> > isn't fixed yet. But I'd be interested to know if you have any thoughts
> > on what kind of time-frame this would take.
> 
> I am finding it increasingly difficult to specify this, since I
> haven't done a project this big before. A lot of it would depend on
> which parts of my application design you agree to. I had thought that
> implementing each of them could take two weeks each and then I could
> look amqp or celery+rabbitmq or others. Now I think ensuring backward
> compatibility would also add about two to three weeks into the mix
> leaving very little time for me to add other backends.

So, this shows that you have thought about the break down of this time,
and how long things will take, even if the estimates turn out to be
wildly incacurate. I think listing specific milestones here, from your
break down would be useful, even if they end up changing, as the design
is refined.

  Tom