[Buildbot-devel] Problems getting svnpoller.py to work

[Brian Warner]

Robert Mecklenburg <rmecklenburg at s5w.com> writes:

> I'm new to buildbot and I'm having trouble configuring it.  My set up
> is ubuntu/edgy my master and slave on the same host (using different
> accounts).  I've got them setup so a forced build works, but when I
> try to get buildbot to initiate a build triggerred by an svn checkin
> I'm having problems.

Ah. It took me a couple of tries to understand the problem here.. but you're
right, the way the buildbot currently handles branches isn't going to allow
you to use an unmodified svnpoller.py with a repository file-naming
convention/policy that puts the branch name before the project name.

First, some background:

  As you noticed, there are a couple of places in buildbot where branch=None
  is used to indicate that we're building the trunk. There are several pieces
  that need to agree on this convention, however.
  
  The first part is to make sure your ChangeSource is reporting branch
  information the way you like it. contrib/svnpoller.py will only send
  Changes with branch="trunk" (this is expressed on line 81, where it adds
  --branch=trunk to the 'buildbot sendchange' command line). In addition, all
  the filenames that it sends are relative to the top of the repository
  (which may begin with /trunk/s5w/, or /s5w/trunk/, or whatever, depending
  upon your SVN repository conventions), whereas for the actual build you
  probably care about filenames relative to the top of your tree (removing
  the /trunk/s5w/ prefix from each one).
  
  To match the filename, you set the PBChangeSource's prefix= to the string
  you want to strip away. To match the --branch=trunk, you would have to set
  the Scheduler's branch= argument to "trunk". With these two pieces, builds
  will be run using a SourceStamp that uses branch="trunk" and a revision
  number equal to the highest number seen in any of the component Changes.
  
  The SVN checkout step must be configured to either handle arbitrary
  branches or not use branches at all. As described in the docs for the SVN
  step (http://buildbot.sourceforge.net/manual-0.7.5.html#SVN), you handle
  arbitrary branches by providing baseURL=, or you avoid using branches
  altogether by providing just svnurl= .

Now, in this case, both SVN options cause problems. If you use branches (by
providing baseURL=), the SVN step will simply concatenate your baseURL= and
the SourceStamp's branch attribute to come up with an svn repository URL.
This only works for a repository filenaming policy that puts the project name
first, followed by the branch name (and then followed by the source file
name). For your repository's policy, it would create something like
svn://host/repo/trunk/s5w/trunk/, which is obviously wrong.

On the other hand, the svnurl= mode (which doesn't use branches) has that
annoying 'assert not branch' that you identified. The idea was that if you're
using svnurl=, you shouldn't be passing it Changes or SourceStamps that
suggest you *do* want to use a branch, because that stands a good chance of
losing information and checking out the trunk where you really wanted to
check out a branch.

So I think the simplest path I would suggest for you (involving the fewest
changes) would be to stick to branch=None to mean "trunk" everywhere. In this
case, that means starting with svnpoller.py: just remove the
--branchname=trunk option from line 81. That will give you Changes that have
branch=None. Then you will need to have a Scheduler with branch=None to
match. Your SVN checkout step should then use svnurl= , with a value that
ends in "/trunk/s5w" (since it will be used directly in the 'svn checkout'
command, with nothing appended to it).

So do this:

 c['sources'].append(PBChangeSource(prefix="/trunk/s5w/"))
 c['schedulers'].append(Scheduler(name="trunk", branch=None,
                                  treeStableTimer=10,
                                  builderNames=["s5-full"]))
 ...
 f.addStep(source.SVN, svnurl="svn://host/repo/trunk/s5w")

and change svnpoller.py:81 from:
   cmd = "buildbot sendchange --master="+buildmaster+" --branch=trunk --revision=\""+revision+"\" --username=\""+author+"\" --comments=\""+comments+"\" "+" ".join(paths)
to:
   cmd = "buildbot sendchange --master="+buildmaster+" --revision=\""+revision+"\" --username=\""+author+"\" --comments=\""+comments+"\" "+" ".join(paths)


The root cause of the problem is that svnpoller.py shouldn't be passing
--branch=trunk as the default.. it should probably leave the branch out (and
thus using branch=None) if it doesn't know any better. BTW, svnpoller.py has
been deprecated in 0.7.6, having been replaced with the in-buildmaster
buildbot.changes.svnpoller module (see the docs in
http://buildbot.sourceforge.net/manual-0.7.5.html#SVNPoller), which provides
better support for branchnames, and doesn't require an extra cron command.

The secondary problem is that the source.SVN step doesn't currently know how
to combine a baseURL, a branch name, and a project name properly when your
repository is using the branchname/projectname/filename convention. If we had
a way to express this, then we could use branch="trunk" instead of
branch=None, and your setup could handle arbitrary named branches instead of
being limited to the trunk as the suggestions I've listed above would do.


Eventually I think we need to pass around instances of some Branch class
rather than just strings, and enhance the SVN checkout step to be able to ask
the Branch object how to combine the branch and the repository names
properly. When we enhance the status displays to present multiple-branch
builds better, we'll probably make a pass through the checkout steps at the
same time.

hope that helps,
 -Brian