[devel at bb.net] Latent SSH Builder

[Thomas Heller]

Hi all,

I wanted to probe interest/opinions/directions for a possible
implementation of running the buildsteps over a latent SSH connection to
a given server.

First of all, I want to give my motivation why I think I need this
specific feature:
We (https://github.com/STEllAR-GROUP/hpx) develop a parallel runtime
system and obviously need to test on various platforms and different
configuration, this includes larger clusters where we are not admins and
running a daemon such as buildslave is commonly frowned upon and
potentially creates a massive security hole as those systems are usually
shared among hundreds of users. Those systems are usually just accessed
over SSH. Once you have logged on, you submit a batch job and do your
thing. The nodes the batch jobs run on don't have to have an outgoing
internet connection (most of them aren't even NATed). Starting a
buildslave on those login nodes is generally possible, however, I would
find it more elegant if we could create a latent slave that initiates
the SSH connection and then executes the buildsteps inside that SSH session.
I hope this description makes sense.

The most obvious solution might be to just prefix the commands in the
buildsteps with "ssh <host>". While this would certainly work, it feels
like it is doing too much, in that scenario, we need to initiate a ssh
connection for each new buildstep. In addition, steps like source code
checkout of a, for example, git repository wouldn't fit into that scheme

A solution to that problem that comes to my mind after browsing through
the buildbot docs and code is to combine the LocalBuildSlave with a
LatentBuilder. Here comes the problem ... I have no idea where to start.
So far I am at the stage of starting to understand how the command
invocation works for the BuildSlaves, in the case of the
LocalBuildSlaves it just circumvents the whole network protocol of
pickling/unpickling and invokes the command directly (please correct me
if I am wrong). Here comes the problem though, instead of invoking just
yet another process, I need to invoke command in my previously initiated
SSH connection and at the moment I fail how to add this specific
functionality.

Any ideas and further directions are highly appreciated!

Cheers,

Thomas