[Buildbot] #2437: New Master-Slave Protocol
Buildbot trac
trac at buildbot.net
Sun Feb 1 13:06:21 UTC 2015
#2437: New Master-Slave Protocol
------------------------+-----------------------
Reporter: dustin | Owner: tardyp
Type: enhancement | Status: assigned
Priority: major | Milestone: 0.9.+
Version: 0.8.7p1 | Resolution:
Keywords: |
------------------------+-----------------------
Changes (by tardyp):
* owner: => tardyp
Old description:
> Buildbot currently uses Twisted Python's ''perspective broker'' to
> communicate with slaves. This is a remote procedure call library that
> operates on TCP connections, and has a few disadvantages:
> * It is Python-only, so slaves must be implemented in Python
> * It keeps the TCP connection open for many hours, and does not handle
> connection failure well
> * The RPC model is complex and does not map well to the operations
> Buildbot performs
> * The RPC implementation is inefficient and imposes some arbitrary
> limits on Buildbot's flexibility.
>
> == Scope ==
> Here be dragons! Several people have attempted this before:
> * http://src.chromium.org/viewvc/chrome/trunk/tools/slavelastic/
> (maruel)
> * http://github.com/djmitche/remsh (djmitche)
> either of these may be a great starting point for this project, but at
> any rate this is a challenging project that will require a lot of
> thoughtful design work. The current plan is to use [http://amp-
> protocol.net amp] over
> [http://twistedmatrix.com/documents/current/conch/index.html ssh] as the
> default slave protocol.
>
> It's probably worth looking at off-the-shelf Message Queuing projects
> like [http://www.zeromq.org/intro:read-the-manual zeromq],
> [http://www.rabbitmq.com/tutorials/tutorial-six-python.html RabbitMQ] or
> [http://ask.github.com/celery/getting-started/introduction.html Celery].
>
> The best approach is to find a way to get some working code put together
> quickly, while still allowing Buildbot to do everything its users expect.
> For example, if you can write your new protocol such that Buildbot can
> use perspective broker ''or'' your protocol, but you only get as far as a
> simple slave that can run {{{echo hello world}}}, that's great! We can
> ship that code, and someone else can pick up where you've left off to add
> more capabilities to your protocol -- assuming your design does not make
> that impossible.
>
> It would be great to select a communication protocol that is not Python-
> specific, so that a non-Python slave could be used to run Buildbot on
> more limited hardware (e.g., mobile devices).
>
> = Remaining Work =
>
> * Stop leaking PB references out of the protocol - in particular,
> startCommand takes a !RemoteCommand argument on which it calls `remote_*`
> methods.
New description:
Buildbot currently uses Twisted Python's ''perspective broker'' to
communicate with slaves. This is a remote procedure call library that
operates on TCP connections, and has a few disadvantages:
* It is Python-only, so slaves must be implemented in Python
* It keeps the TCP connection open for many hours, and does not handle
connection failure well
* The RPC model is complex and does not map well to the operations
Buildbot performs
* The RPC implementation is inefficient and imposes some arbitrary limits
on Buildbot's flexibility.
== Scope ==
Here be dragons! Several people have attempted this before:
* http://src.chromium.org/viewvc/chrome/trunk/tools/slavelastic/ (maruel)
* http://github.com/djmitche/remsh (djmitche)
either of these may be a great starting point for this project, but at any
rate this is a challenging project that will require a lot of thoughtful
design work. The current plan is to use [http://amp-protocol.net amp] over
[http://twistedmatrix.com/documents/current/conch/index.html ssh] as the
default slave protocol.
It's probably worth looking at off-the-shelf Message Queuing projects like
[http://www.zeromq.org/intro:read-the-manual zeromq],
[http://www.rabbitmq.com/tutorials/tutorial-six-python.html RabbitMQ] or
[http://ask.github.com/celery/getting-started/introduction.html Celery].
The best approach is to find a way to get some working code put together
quickly, while still allowing Buildbot to do everything its users expect.
For example, if you can write your new protocol such that Buildbot can use
perspective broker ''or'' your protocol, but you only get as far as a
simple slave that can run {{{echo hello world}}}, that's great! We can
ship that code, and someone else can pick up where you've left off to add
more capabilities to your protocol -- assuming your design does not make
that impossible.
It would be great to select a communication protocol that is not Python-
specific, so that a non-Python slave could be used to run Buildbot on more
limited hardware (e.g., mobile devices).
= Remaining Work =
Stop leaking PB references out of the protocol. grep pb.referenceable
gives a good idea of what's remaining
- process.builder.Builder: <tardyp> thinks this is not needed anymore.
There is a docstring also in that file documenting removed method argument
- process.remotecommand.!RemoteCommand:
protocol.Base.startCommand takes a !RemoteCommand argument on
which it calls `remote_*` methods. Needs to create a proxyRemoteCommand
that forwards the call to the process implementation.
- process.slavebuilder.!AbstractSlaveBuilder:
<tardyp> did not find any reason why this is referenceable
- schedulers.trysched: This is the old pb interface to try commandline.
should be removed as part of #896
- steps.slave , steps.transfer: There is a FileWriter/FileReader
interface used for transferring files, that is passed via the
RemoteCommand args for commands 'uploadDirectory', 'uploadFile' and
'downloadFile'. Should create a proxy of those interfaces
--
Comment:
I'll be trying to pick the low hanging fruits here, as it looks like there
are a lot of needs for better slave protocol.
I've been at least trying to see what's really missing here. please
@dustin, and @tomprince help and tell if I'm wrong.
--
Ticket URL: <http://trac.buildbot.net/ticket/2437#comment:7>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation
More information about the bugs
mailing list