[Buildbot-devel] copying information between steps or schedulers

[Brian Warner]

[sorry I've been so out of touch recently.. the day job's been keeping me
pretty busy]

For this purpose, and some others which have been raised on the list before,
I'm thinking that some sort of general-purpose "build attributes" might be
appropriate. These would be a set of key-value pairs made available to all
Steps, and persisted in the long-term BuildStatus object to be made available
to status plugins. I like the "%(attrname)s" substitution syntax suggested
earlier to get the attribute into the ShellCommand args, although I'm
somewhat tempted to make them more explict, maybe something like:

 s(ShellCommand, args=["mv", "source.tar.gz",
                       SubstituteAttrs("build-r%(revision)s.tar.gz")])

or maybe

 s(ShellCommand, args=["mv", "source.tar.gz",
                       SubstituteAttrs("build-r%ss.tar.gz", "revision")])


I think that you'd still have to write some code to set these attributes, but
could use these %s-substitutions to read them back.

At the moment, the SourceStamp used for the build is a special-purpose
attribute. It might make sense to generalize it in this same fashion. One
problem is that SourceStamp actually contains four pieces (branch, revision,
patch, changes), and that for Change-triggered builds, the revision is not
known until a VC-specific Source step has examined the Changes and decided
what exactly should be checked out. (and for forced builds of HEAD, the
buildbot never actually learns what revision is used, unless we add some code
to parse the output of the checkout/update step somehow).


In general, I'm of two minds on this sort of thing. When I'm wearing my
"Continuous Integration" hat, I want to see the buildbot know as little as
possible about the build process. In my mind, developers should only need to
know how to do "make all; make test", or something equivalent, and the rest
of the intelligence should be built into the tree somewhere, in the top-level
Makefile, or scons script, or Ant recipe, or whatever. This is important
because if it isn't checked in, it will be hard to keep in sync, hard to
maintain, and eventually forgotten. From this point of view, the buildbot
should be as dumb as possible.

On the other hand, when I'm wearing my "Release Engineer" hat, I want to see
the buildbot help with release automation, which frequently involves more
complicated steps that those needed by mere developers. If the buildbot is a
useful place to build installers/.deb-packages, upload them to a distribution
directory, rebuild docs, update a website.. then sure, go for it.

So these two views are sometimes in conflict. I'm tempted to add more
complicated features to the buildbot, to enable the automation features, but
I'm also afraid that if they are available, people will be tempted to put the
build intelligence in the buildbot's config, where it is generally invisible
to developers who may need to do the same thing too.


Anyway, the build attributes are an example of a feature that I'm inclined to
put in, but am still concerned that it might be a bad idea in the long run.
The other feature I'm considering that falls into this category is a pair of
BuildSteps that move files between the master and the slave. The syntax would
probably be something like:

 s(CopyToSlave, source="foo", dest="foo")
 s(CopyFromSlave, source="dist/foo.tar.gz", dest="foo.tar.gz")

where all filenames are relative to the master/slave's respective basedirs.
It would probably be useful to add a third command:

 s(MasterShellCommand, args=["scp", "foo.tar.gz", "website:latest.tar.gz"])

so you could actually do something with the file once you'd copied it to the
master.


thoughts?
 -Brian