[Buildbot-devel] Running build slaves on the sourceforge compile farm

[Olly Betts]

Hi,

I'm currently using tinderbox to run test builds for the Xapian search
library:

http://www.oligarchy.co.uk/tinderbox/xapian/status.html

Most of the builds are run on machines in the sourceforge compile farm.

I'm looking at moving the setup to buildbot, as it looks like I can
get it to work as I want a lot more easily (in particular being able to
customise by subclassing rather than rewriting chunks will make it
possible to keep my customisation separate).  And I'll get useful
features like email notification which I don't currently get with
tinderbox.

I've read most of the information on the website, and it seems to me
that the tricky part is going to be the build slaves on the compile farm
hosts.

You can't make connections to the outside world from the compile farm,
and you can't even reliably send mail from it (which is how tinderbox
expects to work).  It's also worth noting that cron is often unreliable
too.

They all mount an NFS mounted home directory, and the quota is tight, so
I have one source tree, and use VPATH builds so they can share it.

You can only connect in with ssh, and all access has to go via one
machine (cf-shell).  So to connect to one of the MacOS X boxes, you set
up the ssh public/private keys and do:

ssh cf-shell.sf.net ssh ppc-osx1

My current approach runs the build scripts over ssh from an external
box, with stdout and stderr returning over the ssh connections.  Then
the tinderbox mails are sent from the external box rather than from
the machines actually doing the building.

What I've read suggests I can't run the usual buildslaves on the
compilefarm hosts, as they expect to make a TCP connection to the
buildmaster - a good design for a more typical firewall, but unhelpful
here.

I wonder if I can use the external box as a proxy buildslave.  It
could (on cron say) probe each compilefarm host.  If it can connect,
the external box contacts the buildmaster on behalf of the compile farm
host and build commands get executed over ssh.  If this avoided me
having to install Twisted on every compilefarm host I wanted to use,
that would be a bonus.

Would the scheme I describe above work?

Or can you suggest a better way to go about this?

Cheers,
    Olly