[Buildbot-devel] GSoC 2012 proposal for Implementation of a new Master-Slave Protocol.

[Saurabh Kumar]

Hello,

I would be glad to receive comments on this draft of my proposal.

------------------------------------------------------------------------------------

Name: Saurabh Kumar
e-mail: saurabhgeek92 at gmail.com
irc nick: sa1 on freenode.net

Project Proposal:

Implementing a new master-slave protocol:

While the exact nature of the protocol would evolve over the project
timeline, the protocol should have the following features:
- It would be a message-based protocol, rather than a stream or rpc
based protocol.
- It would be a thin abstraction layer over other protocols, so that
multiple backend protocols can be added.
- The data to be transferred should be serialized through JSON. I also
considered Protobuf and Apache Thrift, but I believe that there is no
need for a binary protocol, which would add complexity.
- The layer should be thin enough so that backend implementations are
not limited by the abstraction over them.
- The backends I am looking at adding are amp, 0mq, amqp. The
abstraction layer should support easy addition of backends, so that
the user has a choice.
- The connections should optionally be on a secure protocol, such as
SSH or TLS. This seems to be the least effort way to add security till
backends add their own security features. There would be implications
on what it means for the backend since the backends and the tunnel may
handle reconnections, etc differently.
- The protocol itself should be scalable to multiple masters, to a
large number of slaves, etc. and should gracefully handle the removal
of masters or slaves due to disconnection or other problems. It is to
be seen how  e.g 0mq's brokerless design(or any other fitting design)
can be ported to or will be compatible with the higher level
abstraction.
-Backward compatibility would somehow have to be maintained, so that
current masters work with old slaves, or current slaves with old
masters. At the very least, the user should be able to choose between
the present protocol and the new one.

A schedule for the above would ideally be:
1) Understanding the code, coming up with network models and
implementing toy models of proposed protocols can be done in the
period before coding starts. This will also be the period that I will
have to learn and compensate for any missing experience, including the
nature of asynchronous programming, Twisted, the mq libraries, and
with development processes, testing etc in general.
2) Ideally when the coding period starts, it should be apparent about
what network models can be implemented across backends, and the nature
of the abstraction would be clear. Hopefully 2-3 weeks would be enough
to implement the abstraction layer. I may choose to begin with
existing work, such as [1]
3) The rest of the period would be dedicated to writing backends,
starting with amp or 0mq and proceeding further as time allows, while
making required changes to the abstraction layer.
4) Testing and Documentation would be worked upon simultaneously.

If I am not able to complete the project to my or the organizations
satisfaction within the deadline, I would be happy to keep working on
it. I would be happy to do other work or related work after the
application period too.

Experience:
I am new to open source development, though I have been observing it
from outside for some time. I am familiar with python but am new to
twisted, mq libraries, and asynchronous systems. I have worked with
simple distributed applications before. I hope to learn them as part
of my summer work. I am used to simple workflows on git. I am a second
year student of IIT Kharagpur studying Mathematics and Computing, and
have tried diverse unrelated projects while trying to choose an area
of interest. These include a versioning filesystem, available on my
github page[2], and on implementing AI for the robocup project[3]. I
am new to opensource so I am not affiliated to any project yet, so I
will try to maintain a instance to test projects some of the hackers
from my local LUG[4] hack on, such as git.git. I will probably also
need hosting help.

[1]: https://github.com/djmitche/buildbot/commits/mq
[2]: https://github.com/sa1
[3]: http://www.robocup.org/
[4]: https://github.com/klug


--Saurabh