[Buildbot-devel] Re: Buildbot-devel digest, Vol 1 #33 - 3 msgs

[Brian Warner]

From: Ed Hartnett <ed at unidata.ucar.edu>
> Well I got it working, but I had to mess around with workdir to do it,
> and I don't know why.
> 
> What is workdir anyway? Who sets it, ultimately?

There are a few different directories used by the buildslave:

 the twistd starting directory: (the current directory when you invoke
 twistd)

  This is where twistd puts twistd.log and twistd.pid . I usually put the
  Makefile with the start/stop targets here too.

 basedir: (set when you create the slave's .tap file)

  This is the slave's base directory. The slave will chdir() here when twistd
  is started, before doing anything else. As long as you don't put ../.. in
  anything, the slave will never touch files outside this directory. This is
  an absolute pathname, usually ~slaveuser/buildslave or something like that.

  I usually make this the same as the twistd starting directory.

 builddir: (set in the master.cfg file, as the third element of one of the
 tuples you put in BuildMasterConfig['builders']

  This is a subdirectory of the basedir, under which everything for that
  particular *builder* is placed (there are multiple builders per slave). It
  is also used by the buildmaster as a per-builder subdirectory, as a backing
  store for status logs that haven't been accessed in a long time. Because of
  this, the builddir must be unique across all builders that the master knows
  about.

 workdir: (provided as an argument to most BuildSteps)

  This is a subdirectory of the builddir. The buildslave will start each
  child process (except CVS!) inside this directory (it does a chdir just
  after doing fork). This is generally where the code actually sits. This is
  the directory where you would want to run 'make all'.

  The CVS command puts the checked-out ready-to-compile source in this
  directory. That means it has to run the 'cvs checkout' command from the
  parent directory (builddir), telling CVS to place the checked-out tree in
  'workdir'. The primary reason that this is a subdirectory (instead of
  building directly in builddir) is to make it easy to delete the tree before
  checking out a new one.

  Note that 'cvsmodule' and 'workdir' are independent, and the CVS command
  always specifies '-d' to put the output in workdir. This overrides CVS's
  default behavior of effectively copying the cvsmodule parameter into a '-d'
  argument.

  workdir defaults to 'build', and there's no real use to changing it.

 copydir: (provide as an optional argument to the CVS command)

  This is also a subdirectory of the builddir. If set, the CVS step will
  checkout the tree into it, and then copy the whole thing over into workdir.
  This lets you do an efficient 'cvs update' on your source tree (to minimize
  network bandwidth for remote repositories), but still build a clean tree
  each time.

  The basic build factories have a 'cvsCopy' argument, and if it is set then
  copydir is set to use 'original'


So if we have:

 twistd starting directory = ~buildslave/slaveA
 basedir = ~buildslave/slaveA
 builddir[0] = 'A-full'
 workdir[0] = 'build'
 (and possible builddir[1] = 'A-quick', workdir[1] = 'build', etc)

Then the top-level Makefile and ./configure script will be in:

 ~buildslave/slaveA/A-full/build/Makefile
 ~buildslave/slaveA/A-full/build/configure

The CVS command used to checkout the tree will be:

 cd ~buildslave/slaveA ; cvs -d $REPOSITORY checkout -d workdir $CVSMODULE

(if cvsCopy is set, that will be '.. checkout -d copydir $CVSMODULE',
followed by 'cp -r copydir workdir')
 
> Is it the top level under which the cvs calls get the module? In whcih
> case, workdir + "/" + cvsmodule is where the ./configure must be run,
> not workdir...

Nope, because we always use -d to override this behavior. I did this so that
you wouldn't have to provide the name of the CVS module to all the other
commands.

> class TaggedBuildFactory(ConfigurableBuildFactory):
>     def __init__(self, cvsroot, cvsmodule, tag, configure="configure", configureEnv={}, compile="make all",
>                  test="make check", workdir="."):
>         steps = []
>         steps.append((RmCommand, {'workdir': workdir,
>                                      'command': "rm -rf "+cvsmodule}))
>         steps.append((ShellCommand, {'workdir': workdir,
>                                      'command': "cvs -d "+cvsroot+" co -r "+tag+" "+cvsmodule}))
>         new_workdir = workdir + "/" + cvsmodule
>         steps.append((Configure, {'workdir': new_workdir,
>                                   'command': configure,
>                                   'env': configureEnv}))
>         steps.append((Compile, {'workdir': new_workdir,
>                                 'command': compile}))
>         steps.append((Test, {'workdir': new_workdir,
>                              'command': test}))
>         self.steps = steps
> 
> In master.cfg:
> 
> f_all = TaggedBuildFactory(repository, cvsmodule, "netcdf-3_5_1-beta13",
>                           configure="./configure",
>                           configureEnv={'FC': 'g77', 'CXX':'g++', 'CPPFLAGS':'-Df2cFortran', 'F90':''},
>                           test="make test", workdir="/home/ed/BuildBot/rodney_all")

I think that would work, but I'd do it this way:

class TaggedBuildFactory(ConfigurableBuildFactory):
    def __init__(self, cvsroot, cvsmodule, tag,
                 configure="configure", configureEnv={},
                 compile="make all",
                 test="make check",
                 workdir="build"):
        steps = []
        steps.append((RmCommand, {'workdir': workdir,
                                  'command': "rm -rf %s" % workdir}))
        cvs = "cvs -d %s co -r %s -d %s %s" \
              % (cvsroot, tag, workdir, cvsmodule)
        steps.append((ShellCommand, {'workdir': workdir,
                                     'command': cvs}))
        steps.append((Configure, {'workdir': workdir,
                                  'command': configure,
                                  'env': configureEnv}))
        steps.append((Compile, {'workdir': workdir,
                                'command': compile}))
        steps.append((Test, {'workdir': workdir,
                             'command': test}))
        self.steps = steps

In addition, master.cfg should set the workdir= to be a relative directory.
Just leave it at "build". If you use an absolute pathname there, you won't be
able to move the buildslave's basedir without updating the buildmaster's
configuration. (In fact it's a bug that you're able to pass an absolute
pathname.. the buildslave admin decides where the slave gets to run, not the
buildmaster admin. It should not be easy for the buildmaster to just drop
files anywhere on the slave host that it pleases.)

(of course the buildmaster *can* do whatever is pleases [unless you run the
slave in a chroot], but it shouldn't be *easy* :)

cheers,
 -Brian