[Buildbot-devel] slave connection over ssl possible?

[Brian Warner]

exarkun at twistedmatrix.com wrote:
> 
> I think the biggest argument against this is that buildbot doesn't
> really need any of the features Foolscap provides that PB doesn't. In
> fact, buildbot hardly even needs any of the features that PB provides
> that, say, AMP or XML-RPC don't.

Yeah, that's fair. Buildbot's RPC layer is currently expressed in terms
of object references and remote methods (for example, the buildslave
gets a RemoteReference to the master-side RemoteCommand instance, and
sends stdout/stderr/rc values directly to its remote_status method).
This ties the build process rather tightly to a specific TCP connection.
If I were doing it all over again, I'd probably eschew object references
(just like I'm currently doing away with the status pickles) and have
the slave send more generic status updates indexed with a build number /
step number / command number, or some other unique identifier. This
would let the build tolerate connection loss+reconnect more easily.

> One of the claims I often hear leveled against buildbot (unfairly, I
> think, but there you go) is the difficulty of setting up new slaves. I
> think that dealing with a FURL would be fodder for this sort of
> argument.

Hm. I think that needing to install pyopenssl and foolscap would
certainly be a drag. But I think configuration (at least from the slave
admin's point of view) would actually be easier: one string to deal with
instead of four.

>From the master admin's point of view, I think the change would be
neutral. In the current design they have to find out what their hostname
is, choose+configure a free TCP port, choose+configure slave name and
password, then assemble these four items and hand them to the slave
admin. In a foolscap-based design they'd find+configure the hostname,
let foolscap pick a TCP port for them (or configure it), and
choose+configure the slave name. Then they'd be told the FURL to hand to
each slave admin.

I know that a buildslave admin on one of my masters needs to relearn
their slave's contact information, it takes me a few minutes to locate
all the pieces, especially because e.g. the tahoe buildmaster has them
spread out among multiple files. Looking in a single file named
"buildslave.furls" or something would make that task easier.

But yeah, actual difficulty versus perceived difficulty is a big gap,
and it's in our interest to not feed the detractors.

> When you were thinking about this, what was your motivation?  What does 
> Foolscap bring to buildbot?

I've had a couple of more security-minded folks complain about the
unencrypted slave connection, particularly because there's a password
involved. (telling them that the buildslave password is really there
just to discourage block-the-real-slave nuisances doesn't seem to
mollify them). I know people who've wanted to use the buildbot on
closed-source projects and send e.g. SVN username/password to the
buildslaves, but were worried about who might be able to see them. And I
can imagine closed-but-distributed projects that want to run buildslaves
outside of their secure LAN and not expose their code to anyone else,
who would be worried both about traffic on the wire and false slaves
pretending to be real ones to get access to repository information
and/or credentials.

On the code side, If we could switch completely away from PB to
Foolscap, the buildmaster code would become simpler (the Dispatcher and
the assorted Realm/Avatar/Mind stuff would go away), removing maybe 100
or 200 lines of code. And the ReconnectingPBClientFactory class and all
of pbutil.py would go away (since that's all built in to Foolscap).

So, it wouldn't be a huge win, but if I were asked to provide "slave
connection over ssl" like the Subject says, this is how I'd do it.

cheers,
 -Brian