[Buildbot-commits] [Buildbot] #2437: New Master-Slave Protocol

[Buildbot trac]

#2437: New Master-Slave Protocol
-------------------------+--------------------
Reporter:  dustin        |       Owner:
    Type:  project-idea  |      Status:  new
Priority:  major         |   Milestone:  0.9.+
 Version:  0.8.7p1       |  Resolution:
Keywords:                |
-------------------------+--------------------

Old description:

> Buildbot currently uses Twisted Python's ''perspective broker'' to
> communicate with slaves.  This is a remote procedure call library that
> operates on TCP connections, and has a few disadvantages:
>  * It is Python-only, so slaves must be implemented in Python
>  * It keeps the TCP connection open for many hours, and does not handle
> connection failure well
>  * The RPC model is complex and does not map well to the operations
> Buildbot performs
>  * The RPC implementation is inefficient and imposes some arbitrary
> limits on Buildbot's flexibility.
>
> == Scope ==
> Here be dragons!  Several people have attempted this before:
>  * http://src.chromium.org/viewvc/chrome/trunk/tools/slavelastic/
> (maruel)
>  * http://github.com/djmitche/remsh (djmitche)
> either of these may be a great starting point for this project, but at
> any rate this is a challenging project that will require a lot of
> thoughtful design work. The current plan is to use [http://amp-
> protocol.net amp] over
> [http://twistedmatrix.com/documents/current/conch/index.html ssh] as the
> default slave protocol.
>
> It's probably worth looking at off-the-shelf Message Queuing projects
> like [http://www.zeromq.org/intro:read-the-manual zeromq],
> [http://www.rabbitmq.com/tutorials/tutorial-six-python.html RabbitMQ] or
> [http://ask.github.com/celery/getting-started/introduction.html Celery].
>
> The best approach is to find a way to get some working code put together
> quickly, while still allowing Buildbot to do everything its users expect.
> For example, if you can write your new protocol such that Buildbot can
> use perspective broker ''or'' your protocol, but you only get as far as a
> simple slave that can run {{{echo hello world}}}, that's great!  We can
> ship that code, and someone else can pick up where you've left off to add
> more capabilities to your protocol -- assuming your design does not make
> that impossible.
>
> It would be great to select a communication protocol that is not Python-
> specific, so that a non-Python slave could be used to run Buildbot on
> more limited hardware (e.g., mobile devices).

New description:

 Buildbot currently uses Twisted Python's ''perspective broker'' to
 communicate with slaves.  This is a remote procedure call library that
 operates on TCP connections, and has a few disadvantages:
  * It is Python-only, so slaves must be implemented in Python
  * It keeps the TCP connection open for many hours, and does not handle
 connection failure well
  * The RPC model is complex and does not map well to the operations
 Buildbot performs
  * The RPC implementation is inefficient and imposes some arbitrary limits
 on Buildbot's flexibility.

 == Scope ==
 Here be dragons!  Several people have attempted this before:
  * http://src.chromium.org/viewvc/chrome/trunk/tools/slavelastic/ (maruel)
  * http://github.com/djmitche/remsh (djmitche)
 either of these may be a great starting point for this project, but at any
 rate this is a challenging project that will require a lot of thoughtful
 design work. The current plan is to use [http://amp-protocol.net amp] over
 [http://twistedmatrix.com/documents/current/conch/index.html ssh] as the
 default slave protocol.

 It's probably worth looking at off-the-shelf Message Queuing projects like
 [http://www.zeromq.org/intro:read-the-manual zeromq],
 [http://www.rabbitmq.com/tutorials/tutorial-six-python.html RabbitMQ] or
 [http://ask.github.com/celery/getting-started/introduction.html Celery].

 The best approach is to find a way to get some working code put together
 quickly, while still allowing Buildbot to do everything its users expect.
 For example, if you can write your new protocol such that Buildbot can use
 perspective broker ''or'' your protocol, but you only get as far as a
 simple slave that can run {{{echo hello world}}}, that's great!  We can
 ship that code, and someone else can pick up where you've left off to add
 more capabilities to your protocol -- assuming your design does not make
 that impossible.

 It would be great to select a communication protocol that is not Python-
 specific, so that a non-Python slave could be used to run Buildbot on more
 limited hardware (e.g., mobile devices).

--

Comment (by dustin):

 You actually mention *four* protocols - AMP, SSH, JSON, and TCP.  I'd like
 to focus on AMP over sockets (so, TCP or TLS) for Buildbot's default
 implementation, but write the specification and implementation in a
 generic fashion that could also be implemented with JSON-over-AMQP or XML-
 over-HTTP or YAML-over-email or whatever a user would like.  So the choice
 of protocol isn't really the question to be focusing on up-front.  Rather,
 let's describe a message-based interaction between master and slave that
 can then be implemented in one of several on-the-wire formats.

 In IRC, you asked
 {{{
 09:10 < marchael> for now we determined protocol for exchanging messages,
 but I still confused about how I should try to implement this without
 breaking existing perspective broker
 }}}
 and this is indeed a difficult question.  In general, the approach should
 be:
  * maintain a PB implementation on the master side, but deprecate it
  * add a new AMP-based implementation on the master side, parallel to the
 PB implementation
  * replace the slave-side PB implementation with an AMP implementation

 The tricky bits will be in maintaining both the existing PB implementation
 (which has complex interactions with lots of classes on the master) and
 the AMP-based implementation (which should avoid much of that mess) in
 parallel.  It's hard to say what that will look like without digging into
 the code.  Tom, do you have some thoughts on that based on last year's
 work?

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2437#comment:3>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation