[Buildbot-devel] Masters behind firewalls?

[Olly Betts]

On Tue, Sep 07, 2004 at 12:11:04PM -0700, Brian Warner wrote:
> Of course, the slaves will have to be able to check out the sources too,
> which may be difficult unless the source repository is outside both
> firewalls. I'm considering adding a source-acquiring BuildStep which would
> perform a source checkout *on the master* and then send a tarball with the
> results down to each slave: chews up a lot of bandwidth (on a system that may
> not have have a lot), but could be useful in certain configurations like that
> one, particularly for small trees.

I'd find something like this very useful, especially if the "transfer the
sources" step could be overridden with my own implementation.

I'd use rsync over ssh for the job, which is perhaps a more generally
useful default than shipping a tarball - the deltas between source
versions will typically be a lot smaller than the full sources and
this approach scales well to larger projects.

In case it's useful, this is how I currently update the remote sources
for my sourceforge tinderbox builder and see if anything has changed:

#!/bin/bash
changed=0
test "x$1" = "x--force" && changed=1
export RSYNC_RSH=ssh
cd
for dir in xapian-core xapian-examples omega xapian-queryserver ; do
  lines=`rsync -v -t -L -r --delete tinderbox/HEAD/"$dir" cf-shell.sf.net:tinderbox/HEAD|wc -l`
  if test $lines -gt 3 ; then
    changed=1
    echo "$dir has changed ($lines lines of rsync output)"
  fi
done
echo "changed = $changed"

The crude test for more than 3 lines of output was the best way I could
find to check if rsync actually updated anything.  If your VCS checkout
step knows this and you can easily track that fact even if the rsync fails,
you can avoid needing this.

> That technique could also help if the sources are not quite available
> publically and there is some secret or SSH key you need to get a copy.

Or to easily allow a "prepare sources" stage - I find it very useful to
be able to install known good versions of tools required to bootstrap the
sources (autoconf, automake, libtool, bison, etc) in one place and know
that these are used for all builds.  Then I can use the same versions
to build releases.

Cheers,
    Olly