[Buildbot-devel] No branch attribute in SVN source step of builder

Markus Blatt markus at dr-blatt.de
Wed Jan 23 23:26:01 UTC 2013


Hi,

I am currently setting up buildbot for a multiple repository project.
In addition I want to have a scheduler and builder that works for
serveral branches. Depending on the changeset the builder should
checkout the correct branch into a branch specific directory.

My idea was to use the branch property during the SVN checkout with
either Interpolate 

factory.addStep(SVN(repourl=Interpolate(svn_root+name+'/%(branch)s/'),
  workdir=Interpolate('.%(branch)s/'+name),
  mode='incremental',codebase=name, haltOnFailure=True))

or WithProperties

factory.addStep(SVN(repourl=WithProperties(svn_root+name+'/%(branch)s/'),
  workdir=WithProperties('.%(branch)s/'+name),
  mode='incremental',codebase=name, haltOnFailure=True)).

The former does not substitute the branch and the whole repository is
checked out. The latter throws an exception as the branch property is not
there (see attached error excerpt).

I assume that I am missing something that is needed to provide the SVN
checkout with the appropriate properties. But somehow I cannot see my
mistake and would highly appreciate some help. Please find my master.cfg
attached for further inspection.

Thanks a lot for your help!

Cheers,

Markus
-------------- next part --------------
2013-01-23 23:05:29+0100 [-] added buildset 20 to database
2013-01-23 23:05:29+0100 [-] starting build <Build runtests_dune-localfunctions> using slave <SlaveBuilder builder='runtests_dune-localfunctions' slave='serverloft-dun
e1'>
2013-01-23 23:05:29+0100 [-] acquireLocks(slave <BuildSlave 'serverloft-dune1'>, locks [])
2013-01-23 23:05:30+0100 [-] starting build <Build runtests_dune-localfunctions>.. pinging the slave <SlaveBuilder builder='runtests_dune-localfunctions' slave='server
loft-dune1'>
2013-01-23 23:05:30+0100 [-] sending ping
2013-01-23 23:05:30+0100 [Broker,0,188.138.124.154] ping finished: success
2013-01-23 23:05:30+0100 [-] <Build runtests_dune-localfunctions>.startBuild
2013-01-23 23:05:30+0100 [-] Unhandled error in Deferred:
2013-01-23 23:05:30+0100 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/buildstep.py", line 587, in _startStep_2
            d = self.build.render(getattr(self, renderable))
          File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/properties.py", line 191, in render
            return props.render(value)
          File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/properties.py", line 151, in render
            return defer.maybeDeferred(renderable.getRenderingFor, self)
        --- <exception caught here> ---
          File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 125, in maybeDeferred
            result = f(*args, **kw)
          File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/properties.py", line 298, in getRenderingFor
            s = self.fmtstring % pmap
          File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/properties.py", line 258, in __getitem__
            rv = properties[key]
          File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.7p1-py2.6.egg/buildbot/process/properties.py", line 79, in __getitem__
            rv = self.properties[name][0]
        exceptions.KeyError: 'branch'
        
-------------- next part --------------
# -*g python -*-
# ex: set syntax=python:

# This is a sample buildmaster config file. It must be installed as
# 'master.cfg' in your buildmaster's base directory.

# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}

svn_root = 'https://svn.dune-project.org/svn/'
modules =  ['dune-common', 'dune-istl', 'dune-geometry', 'dune-grid', 'dune-localfunctions', 'dune-grid-howto', 'dune-grid-dev-howto']
branches_to_build = [None, 'branches/release-2.2']

codebaseDependencies ={
		    'dune-common': ['dune-common'],
		    'dune-istl': ['dune-common','dune-istl'],
		    'dune-geometry': ['dune-common', 'dune-geometry'],
		    'dune-grid': ['dune-common', 'dune-geometry', 'dune-grid'],
		    'dune-localfunctions': ['dune-common', 'dune-geometry', 'dune-grid', 'dune-localfunctions'],
		    'dune-grid-howto': ['dune-common', 'dune-geometry', 'dune-grid', 'dune-istl', 'dune-grid-howto'],
                    'dune-grid-dev-howto': ['dune-common', 'dune-geometry', 'dune-grid', 'dune-grid-dev-howto'],
}

####### BUILDSLAVES

# The 'slaves' list defines the set of recognized buildslaves. Each element is
# a BuildSlave object, specifying a unique slave name and password.  The same
# slave name and password must be configured on the slave.
from buildbot.buildslave import BuildSlave
c['slaves'] = [BuildSlave("name", "password")]

# 'slavePortnum' defines the TCP port to listen on for connections from slaves.
# This must match the value configured into the buildslaves (with their
# --master option)
c['slavePortnum'] = 9989

####### CHANGESOURCES

# the 'change_source' setting tells the buildmaster how it should find out
# about source code changes.  Here we point to the buildbot clone of pyflakes.

#from buildbot.changes.gitpoller import GitPoller
from buildbot.changes.svnpoller import SVNPoller, split_file_branches 
c['change_source'] = []
#c['change_source'].append(GitPoller(
#        'git://github.com/buildbot/pyflakes.git',
#        workdir='gitpoller-workdir', branch='master',
#        pollinterval=300))
c['change_source'].extend(SVNPoller(
      svnurl=svn_root+name,
      project=name,
      split_file=split_file_branches,
      pollinterval=120,
         ) for i, name in enumerate(modules))

###### SCHEDULERS

# Configure the Schedulers, which decide how to react to incoming changes.  In this
# case, just kick off a 'runtests' build

from buildbot.schedulers.basic import SingleBranchScheduler
from buildbot.schedulers.forcesched import ForceScheduler
from buildbot.changes.filter import ChangeFilter
from buildbot.process.properties import Property, Interpolate, WithProperties
c['schedulers'] = []

def filter_branches(branch):
    return branch in branches_to_build

for name in modules:
    module_codebases={}
    for mod in codebaseDependencies[name]:
        for branch in branches_to_build:
            module_codebases.update({mod: {'repository': svn_root+name,
                                           'branch': branch,
                                           'revision': None},})
    
    c['schedulers'].append(SingleBranchScheduler(
                            name="runtests_"+name,
                            change_filter=ChangeFilter(branch_fn = filter_branches),
                            treeStableTimer=None,
                            builderNames=["runtests_"+name],
			    codebases=module_codebases))

####### BUILDERS

# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
# what steps, and which slaves can execute them.  Note that any particular build will
# only take place on one slave.

from buildbot.process.factory import BuildFactory
from buildbot.steps.source.svn import SVN
#from buildbot.steps.source import SVN
from buildbot.steps.shell import ShellCommand
from buildbot.config import BuilderConfig
c['builders'] = []

for module in modules:
    factory = BuildFactory()
    for name in codebaseDependencies[module]:
        # Here is the problem. Using WithProperties throws an exception
        # as there is now property branch.
	# If use Interpolate instead then branch is substituted with
	# an empty string and the whole repository is checked out.
    	factory.addStep(SVN(repourl=WithProperties(svn_root+name+'/%(branch)s/'), workdir=WithProperties('.%(branch)s/'+name), mode='incremental',codebase=name, haltOnFailure=True))
	#
	# Using the slave-svn with baseURL checks out the correct branch.
	# from buildbot.steps.source import SVN
        #factory.addStep(SVN(baseURL=svn_root+name+'/', defaultBranch='trunk', workdir=Interpolate('.%(branch)s/'+name), mode='update',codebase=name, haltOnFailure=True))
    
    for name in codebaseDependencies[module]:
        factory.addStep(ShellCommand(command=['./dune-common/bin/dunecontrol','--only='+name, 'autogen'], workdir=WithProperties('.%(branch)s/'), haltOnFailure=True))
    	factory.addStep(ShellCommand(command=['./dune-common/bin/dunecontrol','--only='+name, 'configure'], workdir=WithProperties('.%(branch)s/'), haltOnFailure=True))
        factory.addStep(ShellCommand(command=['./dune-common/bin/dunecontrol','--only='+name, 'make'], workdir=WithProperties('.%(branch)s/'), haltOnFailure=True))
        factory.addStep(ShellCommand(command=['./dune-common/bin/dunecontrol','--only='+name, 'make', 'check'], workdir=WithProperties('.%(branch)s/'), haltOnFailure=True))
    c['builders'].append(BuilderConfig(name='runtests_'+module,
	slavenames=['serverloft-dune1'],
	factory=factory))

####### STATUS TARGETS

# 'status' is a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot/status/*.py has a variety to choose from,
# including web pages, email senders, and IRC bots.

c['status'] = []

from buildbot.status import html
from buildbot.status.web import authz, auth

authz_cfg=authz.Authz(
    # change any of these to True to enable; see the manual for more
    # options
    auth=auth.BasicAuth([("serverloft",">O at g#zt0|HSc")]),
    gracefulShutdown = False,
    forceBuild = 'auth', # use this to test your slave once it is set up
    forceAllBuilds = False,
    pingBuilder = False,
    stopBuild = False,
    stopAllBuilds = False,
    cancelPendingBuild = False,
)
c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))

####### PROJECT IDENTITY

# the 'title' string will appear at the top of this buildbot
# installation's html.WebStatus home page (linked to the
# 'titleURL') and is embedded in the title of the waterfall HTML page.

c['title'] = "DUNE"
c['titleURL'] = "http://wwww.dune-project.org"

# the 'buildbotURL' string should point to the location where the buildbot's
# internal web server (usually the html.WebStatus page) is visible. This
# typically uses the port number set in the Waterfall 'status' entry, but
# with an externally-visible host name which the buildbot cannot figure out
# without some help.

c['buildbotURL'] = "http://localhost:8010/"

####### DB URL

c['db'] = {
    # This specifies what database buildbot uses to store its state.  You can leave
    # this at its default for all but the largest installations.
    'db_url' : "postgresql://user:password@localhost/buildbot",
}

####### CODE BASE GENERATOR
all_repositories = {
       r'https://svn.dune-project.org/svn/dune-common' : 'dune-common',
       r'https://svn.dune-project.org/svn/dune-istl' : 'dune-istl',
       r'https://svn.dune-project.org/svn/dune-geometry' : 'dune-geometry',
       r'https://svn.dune-project.org/svn/dune-grid' : 'dune-grid',
       r'https://svn.dune-project.org/svn/dune-localfunctions' : 'dune-localfunctions',
       r'https://svn.dune-project.org/svn/dune-grid-howto' : 'dune-grid-howto',
       r'https://svn.dune-project.org/svn/dune-grid-dev-howto' : 'dune-grid-dev-howto',

 }
 
def codebaseGenerator(chdict):
      return all_repositories[chdict['repository']]
 
c['codebaseGenerator'] = codebaseGenerator





More information about the devel mailing list