[Buildbot-devel] Sharing files (or other info) between slaves

[Greg Ward]

I'm setting up a buildbot for my company, which inherently requires a
multiplatform build.  (We produce some C++ binaries that run on Linux,
some C++ binaries that run on Windows, and a bunch of Java .jar files.
 Thus we have to do some building on Linux and some on Windows.)

My basic idea is to have N Linux buildslaves and N Windows
buildslaves, and set up builds so that the Linux binaries (and
probably the Java stuff) are done on Linux, and the Windows binaries
(and installers) are done on Windows.  (Currently N = 1, but hopefully
it'll grow to 2 or 3 when I've demonstrated that buildbot will
actually work for us.)

For a couple of reasons, I would prefer to only checkout code once on
a Linux buildslave and then transfer the source tree to a Windows
buildslave.  (It means I don't have to port my custom checkout script
to Windows; it guarantees that both slaves will build the same tree;
it means we only have to run a moderately expensive post-checkout
"setup" step once; and it reduces load on our poor CVS server.)

The "obvious" way to do this is to tar up the source tree on slave1
(Linux) and drop it somewhere that an HTTP or FTP server can see, then
on slave2 (Windows) have a build step that fetches the file and
unpacks it.  But that means I have to write a build step for slave2
that runs

 curl http://slave1/build/source-tree.tar ...

which means that slave2 has to "know about" slave1.  As long as N=1,
that'll work but be a gross hack.  But when N=2, and buildbot gets to
pick between slave1 and slave3 for Linux builds and slave2 and slave4
for Windows builds, I'm screwed.

I suppose I could use FileUpload and FileDownload to transfer the
source tree via the buildmaster, but I am doing this for efficiency
after all ... adding 2 more transfer steps won't help.

Also, I know that at some point I'm going to have to copy files
(installers, executables) from the Windows buildslave over to Linux
for final packaging.  Again, I suppose I could use FileUpload and
FileDownload, but a direct transfer between the slaves would be
faster.

Is there a clean way of doing this?  At the very least, all I need is
to get a hostname (eg. "slave1") to the build steps that will run on
the Windows buildslave.

Thanks --

Greg