[Buildbot-devel] No branch attribute in SVN source step of builder
Markus Blatt
markus at dr-blatt.de
Wed Jan 23 23:26:01 UTC 2013
Hi,
I am currently setting up buildbot for a multiple repository project.
In addition I want to have a scheduler and builder that works for
serveral branches. Depending on the changeset the builder should
checkout the correct branch into a branch specific directory.
My idea was to use the branch property during the SVN checkout with
either Interpolate
factory.addStep(SVN(repourl=Interpolate(svn_root+name+'/%(branch)s/'),
workdir=Interpolate('.%(branch)s/'+name),
mode='incremental',codebase=name, haltOnFailure=True))
or WithProperties
factory.addStep(SVN(repourl=WithProperties(svn_root+name+'/%(branch)s/'),
workdir=WithProperties('.%(branch)s/'+name),
mode='incremental',codebase=name, haltOnFailure=True)).
The former does not substitute the branch and the whole repository is
checked out. The latter throws an exception as the branch property is not
there (see attached error excerpt).
I assume that I am missing something that is needed to provide the SVN
checkout with the appropriate properties. But somehow I cannot see my
mistake and would highly appreciate some help. Please find my master.cfg
attached for further inspection.
Thanks a lot for your help!
Cheers,
Markus
-------------- next part --------------
2013-01-23 23:05:29+0100 [-] added buildset 20 to database
2013-01-23 23:05:29+0100 [-] starting build <Build runtests_dune-localfunctions> using slave <SlaveBuilder builder='runtests_dune-localfunctions' slave='serverloft-dun
e1'>
2013-01-23 23:05:29+0100 [-] acquireLocks(slave <BuildSlave 'serverloft-dune1'>, locks [])
2013-01-23 23:05:30+0100 [-] starting build <Build runtests_dune-localfunctions>.. pinging the slave <SlaveBuilder builder='runtests_dune-localfunctions' slave='server
loft-dune1'>
2013-01-23 23:05:30+0100 [-] sending ping
2013-01-23 23:05:30+0100 [Broker,0,188.138.124.154] ping finished: success
2013-01-23 23:05:30+0100 [-] <Build runtests_dune-localfunctions>.startBuild
2013-01-23 23:05:30+0100 [-] Unhandled error in Deferred:
2013-01-23 23:05:30+0100 [-] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 441, in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/buildstep.py", line 587, in _startStep_2
d = self.build.render(getattr(self, renderable))
File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/properties.py", line 191, in render
return props.render(value)
File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/properties.py", line 151, in render
return defer.maybeDeferred(renderable.getRenderingFor, self)
--- <exception caught here> ---
File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 125, in maybeDeferred
result = f(*args, **kw)
File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/properties.py", line 298, in getRenderingFor
s = self.fmtstring % pmap
File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/properties.py", line 258, in __getitem__
rv = properties[key]
File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/properties.py", line 79, in __getitem__
rv = self.properties[name][0]
exceptions.KeyError: 'branch'
-------------- next part --------------
# -*g python -*-
# ex: set syntax=python:
# This is a sample buildmaster config file. It must be installed as
# 'master.cfg' in your buildmaster's base directory.
# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}
svn_root = 'https://svn.dune-project.org/svn/'
modules = ['dune-common', 'dune-istl', 'dune-geometry', 'dune-grid', 'dune-localfunctions', 'dune-grid-howto', 'dune-grid-dev-howto']
branches_to_build = [None, 'branches/release-2.2']
codebaseDependencies ={
'dune-common': ['dune-common'],
'dune-istl': ['dune-common','dune-istl'],
'dune-geometry': ['dune-common', 'dune-geometry'],
'dune-grid': ['dune-common', 'dune-geometry', 'dune-grid'],
'dune-localfunctions': ['dune-common', 'dune-geometry', 'dune-grid', 'dune-localfunctions'],
'dune-grid-howto': ['dune-common', 'dune-geometry', 'dune-grid', 'dune-istl', 'dune-grid-howto'],
'dune-grid-dev-howto': ['dune-common', 'dune-geometry', 'dune-grid', 'dune-grid-dev-howto'],
}
####### BUILDSLAVES
# The 'slaves' list defines the set of recognized buildslaves. Each element is
# a BuildSlave object, specifying a unique slave name and password. The same
# slave name and password must be configured on the slave.
from buildbot.buildslave import BuildSlave
c['slaves'] = [BuildSlave("name", "password")]
# 'slavePortnum' defines the TCP port to listen on for connections from slaves.
# This must match the value configured into the buildslaves (with their
# --master option)
c['slavePortnum'] = 9989
####### CHANGESOURCES
# the 'change_source' setting tells the buildmaster how it should find out
# about source code changes. Here we point to the buildbot clone of pyflakes.
#from buildbot.changes.gitpoller import GitPoller
from buildbot.changes.svnpoller import SVNPoller, split_file_branches
c['change_source'] = []
#c['change_source'].append(GitPoller(
# 'git://github.com/buildbot/pyflakes.git',
# workdir='gitpoller-workdir', branch='master',
# pollinterval=300))
c['change_source'].extend(SVNPoller(
svnurl=svn_root+name,
project=name,
split_file=split_file_branches,
pollinterval=120,
) for i, name in enumerate(modules))
###### SCHEDULERS
# Configure the Schedulers, which decide how to react to incoming changes. In this
# case, just kick off a 'runtests' build
from buildbot.schedulers.basic import SingleBranchScheduler
from buildbot.schedulers.forcesched import ForceScheduler
from buildbot.changes.filter import ChangeFilter
from buildbot.process.properties import Property, Interpolate, WithProperties
c['schedulers'] = []
def filter_branches(branch):
return branch in branches_to_build
for name in modules:
module_codebases={}
for mod in codebaseDependencies[name]:
for branch in branches_to_build:
module_codebases.update({mod: {'repository': svn_root+name,
'branch': branch,
'revision': None},})
c['schedulers'].append(SingleBranchScheduler(
name="runtests_"+name,
change_filter=ChangeFilter(branch_fn = filter_branches),
treeStableTimer=None,
builderNames=["runtests_"+name],
codebases=module_codebases))
####### BUILDERS
# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
# what steps, and which slaves can execute them. Note that any particular build will
# only take place on one slave.
from buildbot.process.factory import BuildFactory
from buildbot.steps.source.svn import SVN
#from buildbot.steps.source import SVN
from buildbot.steps.shell import ShellCommand
from buildbot.config import BuilderConfig
c['builders'] = []
for module in modules:
factory = BuildFactory()
for name in codebaseDependencies[module]:
# Here is the problem. Using WithProperties throws an exception
# as there is now property branch.
# If use Interpolate instead then branch is substituted with
# an empty string and the whole repository is checked out.
factory.addStep(SVN(repourl=WithProperties(svn_root+name+'/%(branch)s/'), workdir=WithProperties('.%(branch)s/'+name), mode='incremental',codebase=name, haltOnFailure=True))
#
# Using the slave-svn with baseURL checks out the correct branch.
# from buildbot.steps.source import SVN
#factory.addStep(SVN(baseURL=svn_root+name+'/', defaultBranch='trunk', workdir=Interpolate('.%(branch)s/'+name), mode='update',codebase=name, haltOnFailure=True))
for name in codebaseDependencies[module]:
factory.addStep(ShellCommand(command=['./dune-common/bin/dunecontrol','--only='+name, 'autogen'], workdir=WithProperties('.%(branch)s/'), haltOnFailure=True))
factory.addStep(ShellCommand(command=['./dune-common/bin/dunecontrol','--only='+name, 'configure'], workdir=WithProperties('.%(branch)s/'), haltOnFailure=True))
factory.addStep(ShellCommand(command=['./dune-common/bin/dunecontrol','--only='+name, 'make'], workdir=WithProperties('.%(branch)s/'), haltOnFailure=True))
factory.addStep(ShellCommand(command=['./dune-common/bin/dunecontrol','--only='+name, 'make', 'check'], workdir=WithProperties('.%(branch)s/'), haltOnFailure=True))
c['builders'].append(BuilderConfig(name='runtests_'+module,
slavenames=['serverloft-dune1'],
factory=factory))
####### STATUS TARGETS
# 'status' is a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot/status/*.py has a variety to choose from,
# including web pages, email senders, and IRC bots.
c['status'] = []
from buildbot.status import html
from buildbot.status.web import authz, auth
authz_cfg=authz.Authz(
# change any of these to True to enable; see the manual for more
# options
auth=auth.BasicAuth([("serverloft",">O at g#zt0|HSc")]),
gracefulShutdown = False,
forceBuild = 'auth', # use this to test your slave once it is set up
forceAllBuilds = False,
pingBuilder = False,
stopBuild = False,
stopAllBuilds = False,
cancelPendingBuild = False,
)
c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))
####### PROJECT IDENTITY
# the 'title' string will appear at the top of this buildbot
# installation's html.WebStatus home page (linked to the
# 'titleURL') and is embedded in the title of the waterfall HTML page.
c['title'] = "DUNE"
c['titleURL'] = "http://wwww.dune-project.org"
# the 'buildbotURL' string should point to the location where the buildbot's
# internal web server (usually the html.WebStatus page) is visible. This
# typically uses the port number set in the Waterfall 'status' entry, but
# with an externally-visible host name which the buildbot cannot figure out
# without some help.
c['buildbotURL'] = "http://localhost:8010/"
####### DB URL
c['db'] = {
# This specifies what database buildbot uses to store its state. You can leave
# this at its default for all but the largest installations.
'db_url' : "postgresql://user:password@localhost/buildbot",
}
####### CODE BASE GENERATOR
all_repositories = {
r'https://svn.dune-project.org/svn/dune-common' : 'dune-common',
r'https://svn.dune-project.org/svn/dune-istl' : 'dune-istl',
r'https://svn.dune-project.org/svn/dune-geometry' : 'dune-geometry',
r'https://svn.dune-project.org/svn/dune-grid' : 'dune-grid',
r'https://svn.dune-project.org/svn/dune-localfunctions' : 'dune-localfunctions',
r'https://svn.dune-project.org/svn/dune-grid-howto' : 'dune-grid-howto',
r'https://svn.dune-project.org/svn/dune-grid-dev-howto' : 'dune-grid-dev-howto',
}
def codebaseGenerator(chdict):
return all_repositories[chdict['repository']]
c['codebaseGenerator'] = codebaseGenerator
More information about the devel
mailing list