[Buildbot-devel] RFC: "build properties"

[Brian Warner]

[wow, so much traffic on the buildbot-devel list! exciting! but I just wish I
could keep up with all of it..]

I've got about a dozen local branches (kept in darcs repositories on my
laptop) where I'm developing various features and/or evaluating patches and
bugfixes. One of these is to implement "build properties", which I wrote
about a few weeks (months?) ago: the idea is that each build has a dictionary
of "properties", which can be set and retrieved by various BuildSteps, and
which could also be substituted into ShellCommands. The most obvious example
is the revision number of the code currently being built, but there are other
possibilities.

I've attached the section of the user's manual that describes these
properties. I'm looking for comments on this feature before I get too far in
implementing them. Does this seem useful? Maintainable? Making it possible to
use the properties without writing a custom BuildStep seems pretty
straightforward, but I'm concerned about making it possible to *set* the
properties without writing code. (the interface to set_property would
basically be a bunch of regexps that you could pass to ShellCommand to tell
it to extract a string out of the command's stdout, and I'm thinking that
regexps that big are effectively code anyway, and where there's code to be
written, I'd like to encourage people to use a full-scale language for it.
But, I could be completely wrong about that).

I'll be asking for comments on a couple of features like this in the next few
days/weeks. I've been trying to stay in the habit of writing the user's
documentation first, then the unit tests, then actually implementing the
feature, so I've got docs for many new things that are not yet implemented.

thanks,
 -Brian


@node Build Properties,  , Writing New BuildSteps, Simple ShellCommand Subclasses
@subsubsection Build Properties

@cindex build properties

Each build has a set of ``Build Properties'', which can be used by
each BuildStep to modify their actions. For example, the SVN revision
number of the source code being build is available as a build
property, and a ShellCommand step could incorporate this number into a
command which create a numbered release tarball.

Some build properties are set when the build starts, such as the
SourceStamp information. Other properties can be set by BuildSteps as
they operate, for example the various Source steps will set the
@code{got_revision} property to the source revision that was actually
checked out (which can be useful when the SourceStamp in use merely
requested the ``latest revision'': @code{got_revision} will tell you
what was actually built).

In custom BuildSteps, you can get and set the build properties with
the @code{getProperty}/@code{setProperty} methods. Each takes a string
for the name of the property, and returns or accepts an
arbitrary at footnote{Build properties are serialized along with the
build results, so they must be serializable. For this reason, the
value of any build property should be simple inert data: strings,
numbers, lists, tuples, and dictionaries. They should not include
class instances.} object. For example:

@example
class MakeTarball(step.ShellCommand):
    def start(self):
        self.setCommand(["tar", "czf",
                         "build-%s.tar.gz" % self.getProperty("revision"),
                         "source"])
        step.ShellCommand.start(self)
@end example

@cindex WithProperties

You can use build properties in ShellCommands by using the
@code{WithProperties} wrapper when setting the arguments of the
ShellCommand. This interpolates the named build properties into the
generated shell command.

@example
from buildbot.process.step import ShellCommand, WithProperties

s(ShellCommand,
  command=["tar", "czf",
           WithProperties("build-%s.tar.gz", "revision"),
           "source"],
 )
@end example

If this BuildStep were used in a tree obtained from Subversion, it
would create a tarball with a name like @file{build-1234.tar.gz}.

The @code{WithProperties} function does @code{printf}-style string
interpolation, using strings obtained by calling
@code{build.getProperty(propname)}. You can also use python
dictionary-style string interpolation by using the @code{%(propname)s}
syntax:

@example
s(ShellCommand,
  command=["tar", "czf",
           WithProperties("build-%(revision)s.tar.gz"),
           "source"],
 )
@end example

Note that, like python, you can either do positional-argument
interpolation @emph{or} keyword-argument interpolation, not both. Thus
you cannot use a string like
@code{WithProperties("foo-%(revision)s-%s", "branch")}.

At the moment, the only way to set build properties is by writing a
custom BuildStep.

@heading Common Build Properties

The following build properties are set when the build is started, and
are available to all steps.

@table @code
@item branch

This comes from the build's SourceStamp, and describes which branch is
being checked out. This will be @code{None} (which interpolates into
@code{WithProperties} as an empty string) if the build is on the
default branch, which is generally the trunk. Otherwise it will be a
string like ``branches/beta1.4''. The exact syntax depends upon the VC
system being used.

@item revision

This also comes from the SourceStamp, and is the revision of the
source code tree that was requested from the VC system. When a build
is requested of a specific revision (as is generally the case when the
build is triggered by Changes), this will contain the revision
specification. The syntax depends upon the VC system in use: for SVN
it is an integer, for Mercurial it is a short string, for Darcs it is
a rather large string, etc.

If the ``force build'' button was pressed, the revision will be
@code{None}, which means to use the most recent revision available.
This is a ``trunk build''. This will be interpolated as an empty
string.

@item got_revision

This is set when a Source step checks out the source tree, and
provides the revision that was actually obtained from the VC system.
In general this should be the same as @code{revision}, except for
trunk builds, where @code{got_revision} indicates what revision was
current when the checkout was performed. This can be used to rebuild
the same source code later.

@item slavename

This is a string which identifies which buildslave the build is
running on.

@item buildnumber

Each build gets a number, scoped to the Builder (so the first build
performed on any given Builder will have a build number of 0). This
integer property contains the build's number.

@end table