[Buildbot-devel] setting up a builder's workdir

[Brian Warner]

> However, I'm trying to write a builder that postprocesses a set of
> builds previously completed, and trying with a simple
> '(ShellCommand, command='echo "Hello World !"')' already fails
> because the workdir within which the command would be executed
> doesn't exist yet. In fact, the manual refers to a default 'workdir'
> variable that is predefined in the BuildStep class at class-level,
> though I can't seem to find it anywhere.

It looks like the manual is wrong. The default workdir actually comes from
buildbot/process/base.py:165, where it is the Build class that gets a
class-level attribute named .workdir . The default value is "build".

In the Builder, each BuildStep is specified by a two-item tuple of (class,
dictionary of keyword arguments). In base.py:330, where this BuildStep
specification tuple is finally used to construct the BuildStep instance, if
the specification does not already have a workdir= defined, the Build's
default value is added, just as if you had done something like:

 (ShellCommand, command='echo howdy', workdir="build")

All ShellCommands (and most of their derivatives) interpret workdir= as a
directory they should 'cd' into before executing the command. The Source
class (and of course the SVN/CVS/etc subclasses) interpret workdir=
differently: they start at the builder's base directory (equivalent to a
workdir of ".") and then perform a checkout with -d $WORKDIR or equivalent
(thus creating the workdir). When the Source step does an update, it treats
workdir= the same way that ShellCommand does: it cd's into the workdir and
does 'svn update' (or equivalent). When the Source step does a copy, it does
the checkout/update in a sibling directory (named 'source'), cd's up a level
into the builder's base directory, then does a recursive copy into the new
workdir.


So, the short answer is that you want to put workdir="." in the Steps that
you use to create the workspace, and that it will reduce typing if you
arrange for those steps to create a working directory named "build", so that
you don't have to add workdir="something_other_than_build" in the later
steps. Suppose you've got a tarball in some well-known location, and you want
to unpack it, and then run some commands inside the resulting directory.
Furthermore let's suppose this tarball was constructed in such a way that
when you unpack it, the contents wind up in a directory called 'unpacked'.
(also, I'm using one-string ShellCommands here.. really you should use lists
of strings, like command=["tar", "xzf", "/path/to/foo.tar.gz"] instead of the
more terse but more yuck-must-pass-to-/bin/sh-c-can't-handle-spaces
shell-quoting-is-hard form: command="tar xzf /path/to/foo.tar.gz").

 steps = [
   s(ShellCommand, command="tar xzf /path/to/foo.tar.gz", workdir="."),
   s(ShellCommand, command="mv unpacked build", workdir="."),
   s(ShellCommand, command=["make", "things"]) # executed in build/
 ]

Of course, you have to arrange for build/ to be deleted at the beginning of
each build, otherwise the tarball will be unpacked very badly. The whole
reason for using workdirs instead of doing a lot of work directly in the
builder's base directory is to make cleanup easier.

You might also just be able to use:
   s(ShellCommand, command=["mkdir", "-p", "workdir"], workdir=".")
at the beginning.


The longer answer is that the way I wrote this code is kind of stupid.. this
really *should* be a class-level attribute of the various BuildStep classes,
overridable by a the workdir= key in the BuildStep specification tuple, just
like the manual claims it is. Because then if you had a lot of work to do
outside the workdir, you could create a subclass for it:

 class OutsideTheWorkdirShellCommand(ShellCommand):
     workdir="."

and be done with it. But because Build.setupBuild *always* passes in the
workdir= argument, this technique doesn't work. I should have probably done
something like have BuildStep.workdir default to None, and then have that
mean the BuildStep should ask its parent Build for a default value (since it
is important that all Steps for a single Build use the same value, and this
is best kept in the Build rather than as a class-level attribute of the
BuildStep).

(hrm, the "short answer" appears to be longer than the "longer answer". oops)


I'll add this cleanup to the TODO list. For now, hopefully you can have just
a couple of workdir="." steps to get the build/ directory set up, and then do
everything else with the default workdir.

> Is there any predefined BuildStep that I could use that sets up
> my workdir ? It sounds like a frequent enough task to warrant
> to be provided by buildbot itself.

Yes, true, but the vast majority of the build proceses out there start with a
version-control checkout :). Thus far, the buildbot has always been used to
perform some sort of build/test steps inside a directory which was populated
by a checkout. What else could you possibly want to use the buildbot for? :)
:) :)

Let me ask this: what do you need to do to set up the workdir? Would it be
useful to have a step that was defined like this?:

class UnpackTarball(ShellCommand):
    """This step unpacks a named tarball into the working directory.

    @type  filename: string
    @param filename: the pathname of the .tar.gz file that should be unpacked

    @type  tarballdir: string
    @param tarballdir: all files in the tarball are expected to be inside
                       this directory. This step will rename this directory
                       to WORKDIR after the tarball is unpacked.

    @type  workdir: string (optional, defaults to 'build')
    @param workdir: the tarball's top-level directory will be moved here
                    after unpacking
    """

And perhaps a subclass that accepts a URL instead of a filename. And perhaps
both should accept a function which can compute the filename/URL instead of
requiring that it be hard-coded.

Seem useful?

 -Brian