[Buildbot-devel] Inheritance of Properties by Triggerables

[Doug Latornell]

I mentioned a problem I've been having with properties in Triggerable
schedulers in the Buildbot BoF at PyCon, and I worked a bit with Brian on
the first day of the sprints to categorize it better.  But things got crazy
when I got home from PyCon and this is the first chance I've had to get back
to it.

Background:  I'm using buildbot 0.7.10p1 to support the development of a
Fortran 90 code that models the physics and biology of deep estuaries.  The
primary tests are functional tests that compare the results of a run to
previously generated reference results that the researchers have declared to
be good.  Buildbot gets involved in 2 ways: running various test cases in
response to commits, nightly, and weekly; and running various test cases in
a mode where they generate reference output files for comparison to future
tests.  The latter in implemented using buildbot try --diff /dev/null ...

Two schedulers that demonstrate my property inheritance issue:

# Nightly regression tests; only run if there has been a change
# committed in the preceeding 24 hours
regression = Nightly(name='nightly regression tests', branch=None,
                     builderNames=[],
                     properties={'make_refs':False},
                     onlyIfChanged=True,
                     hour=2, minute=0)
# Builders to create ref files for nightly regression tests
make_refs = Try_Jobdir(name='make refs',
                       builderNames=[],
                       properties={'make_refs':True},
                       jobdir='jobdir')

The builders that these schedulers run contain custom ShellCommand
buildsteps that have code in their start() methods like:

        if self.getProperty('make_refs'):
            ShellCommand.start(self)
        else:
            return SKIPPED

The above works just fine.

But things are more complicated than that for this buildbot.  We have 2 F90
compilers to deal with, and one of them is only licensed on one of the
slaves. All of the slaves share disk storage though, so it is fairly simple
to compile on one slave then copy the executable to another slave's build
directory and execute it there.  I implemented that by breaking the builds
into 2 parts: a compile build that does CVS checkout and compile and ends
with a Trigger step that launches a run build via a Triggerable scheduler to
checkout the code, do the run, and compare the results to the reference
output files.

So now we have a build factory that with the final step:

 # Last step of the compile builder is to trigger the run
 # builder
 make_exec.addStep(Trigger(schedulerNames=[triggerable_name]))

and a triggerable scheduler that will respond to that step:

 # Add a triggerable scheduler to run the code
 triggerable = Triggerable(name=triggerable_name,
                           builderNames=[builder_name])
 c_schedulers.append(triggerable)

My expectation was that the make_refs property value that is set in the
Nightly and Try_Jobdir schedulers which use the build factory with the
Trigger step would be inherited by the Triggerable scheduler and thus be
available to its build.  But that doesn't happen.  The build launched by the
Triggerable scheduler fails with an exception.  Here's the traceback:

Traceback (most recent call last):
  File "/home/dlatorne/ocean/lib/python2.4/site-packages/buildbot/process/buildstep.py",
line 687, in startStep
    d.addCallback(self._startStep_2)
  File "/ocean/dlatorne/lib/python2.4/site-packages/twisted/internet/defer.py",
line 195, in addCallback
    callbackKeywords=kw)
  File "/ocean/dlatorne/lib/python2.4/site-packages/twisted/internet/defer.py",
line 186, in addCallbacks
    self._runCallbacks()
  File "/ocean/dlatorne/lib/python2.4/site-packages/twisted/internet/defer.py",
line 328, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
--- <exception caught here> ---
  File "/home/dlatorne/ocean/lib/python2.4/site-packages/buildbot/process/buildstep.py",
line 710, in _startStep_2
    skip = self.start()
  File "/ocean/dlatorne/buildbot/master/extensions.py", line 240, in start
    if self.getProperty('make_refs'):
  File "/home/dlatorne/ocean/lib/python2.4/site-packages/buildbot/process/buildstep.py",
line 637, in getProperty
    return self.build.getProperty(propname)
  File "/home/dlatorne/ocean/lib/python2.4/site-packages/buildbot/process/base.py",
line 224, in getProperty
    return self.build_status.getProperty(propname)
  File "/home/dlatorne/ocean/lib/python2.4/site-packages/buildbot/status/builder.py",
line 1032, in getProperty
    return self.properties[propname]
  File "/home/dlatorne/ocean/lib/python2.4/site-packages/buildbot/process/properties.py",
line 42, in __getitem__
    rv = self.properties[name][0]
exceptions.KeyError: 'make_refs'

So, question #1: Am I right to expect that the make_refs property would be
passed from the Nightly or Try_Jobdir scheduler to the Triggerable
scheduler?  It seems like a reasonable behaviour to me.

I worked around this KeyError exception by using WithProperties to get the
value of the make_refs property in the build factory construction and
setting the property explicitly in the Trigger step:

# Last step of the compile builder is to trigger the run
# builder, with the make_refs property value inherited from the
# scheduler that launched this build.
make_refs = WithProperties('%(make_refs)s')
make_exec.addStep(Trigger(schedulerNames=[triggerable_name],
                          set_properties={'make_refs':make_refs}))

The problem with this is that WithProperties renders the value of make_refs
from a boolean to a string.  To solve that I changed the schedulers to:

# Nightly regression tests; only run if there has been a change
# committed in the preceding 24 hours
regression = Nightly(name='nightly regression tests', branch=None,
                     builderNames=[],
                     properties={'make_refs':'no'},
                     onlyIfChanged=True,
                     hour=2, minute=0)
# Builders to create ref files for nightly regression tests
make_refs = Try_Jobdir(name='make refs',
                       builderNames=[],
                       properties={'make_refs':'yes'},
                       jobdir='jobdir')

and the custom ShellCommand buildstep start() methods to:

        if self.getProperty('make_refs') == 'yes':
            ShellCommand.start(self)
        else:
            return SKIPPED

Question #2:  If the Triggerable can't or shouldn't inherit the make_refs
property and my work around is the correct way to do things, is there a way
to pass the property along without going through the WithProperties.render()
method so that the type of the property values doesn't get mangled (or have
to be managed)?

I guess the inevitable question #3 is: Have I missed the boat somewhere in
all of the above because there is a simpler way of passing the product of
one build to another build?

Doug
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20090416/1ac91799/attachment.html>