[Buildbot-devel] Multiple change sources and scheduler issues
Minesh Patel
mipatel at vmware.com
Tue Jul 1 17:33:33 UTC 2008
Message: 6
Date: Tue, 1 Jul 2008 10:28:06 -0400
From: "Dustin J. Mitchell" <dustin at zmanda.com>
Subject: Re: [Buildbot-devel] Multiple change sources and scheduler
issues
To: "Duncan Ferguson" <duncan.ferguson at altinity.com>
Cc: buildbot-devel at lists.sourceforge.net
Message-ID:
<42338fbf0807010728j5c46a3e4i40cd902c9f4916b6 at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
On Tue, Jul 1, 2008 at 8:34 AM, Duncan Ferguson
<duncan.ferguson at altinity.com> wrote:
> I have set up my buildbot v0.77 with a number of change sources and
> Schedulers. The problem I have is that when a change is detected all
> schedulers are triggered, not just the one configured for the
> particular svn repository.
This is a shortcoming of the current buildbot architecture. The only
workaround I know is to run multiple buildmaster processes. That's
what we do. I've been toying with the idea of running multiple
BuildMaster services inside a single application (by futzing with the
buildbot.tac), but that's a pretty awful hack, and still requires each
slave machine to run a buildslave instance for each buildmaster.
We're about to add a half-dozen additional buildmasters, and each
buildmaster process already has a virtual size of 0.5G, so I'm a
little worried about this issue, too :)
Dustin
Here is a patch for _ONLY_ svn that I have been using successfully. Just how builders are identified in the schedulers, I added a list of how schedulers are identified in the change_sources. Let me know if there are better ideas, I don't mind updating it and submitting a patch but it seems like there would have to be changes to all the 'pollers'. Let me know what you think. Sorry there are no tests yet, but can be added once we figure out if this is the approach we want to take.
In the master.cfg
The schedulerNames would have to be declared on
c['change_source'] = [SVNPoller(svnurl=url,
svnuser=userName,
svnpasswd=userPassword,
schedulerNames=['foo'])]
c['schedulers'] = [Scheduler(name='foo',
branch=None,
treeStableTimer=2*60,
builderNames=[bar])]
Here is the patch:
diff -Nuar buildbot-0.7.7/buildbot/changes/changes.py buildbot-0.7.7-scheduler/buildbot/changes/changes.py
--- buildbot-0.7.7/buildbot/changes/changes.py 2008-03-29 19:30:16.000000000 -0700
+++ buildbot-0.7.7-scheduler/buildbot/changes/changes.py 2008-04-30 09:35:08.000000000 -0700
@@ -205,17 +205,17 @@
d = defer.maybeDeferred(source.disownServiceParent)
return d
- def addChange(self, change):
+ def addChange(self, change, schedulerNames):
"""Deliver a file change event. The event should be a Change object.
This method will timestamp the object as it is received."""
log.msg("adding change, who %s, %d files, rev=%s, branch=%s, "
- "comments %s" % (change.who, len(change.files),
+ "comments %s to schedulers %r" % (change.who, len(change.files),
change.revision, change.branch,
- change.comments))
+ change.comments, schedulerNames))
change.number = self.nextNumber
self.nextNumber += 1
self.changes.append(change)
- self.parent.addChange(change)
+ self.parent.addChange(change, schedulerNames)
# TODO: call pruneChanges after a while
def pruneChanges(self):
diff -Nuar buildbot-0.7.7/buildbot/changes/svnpoller.py buildbot-0.7.7-scheduler/buildbot/changes/svnpoller.py
--- buildbot-0.7.7/buildbot/changes/svnpoller.py 2008-03-29 19:30:17.000000000 -0700
+++ buildbot-0.7.7-scheduler/buildbot/changes/svnpoller.py 2008-04-30 09:42:28.000000000 -0700
@@ -47,7 +47,7 @@
compare_attrs = ["svnurl", "split_file_function",
"svnuser", "svnpasswd",
"pollinterval", "histmax",
- "svnbin"]
+ "svnbin", "schedulerNames"]
parent = None # filled in when we're added
last_change = None
@@ -57,7 +57,7 @@
def __init__(self, svnurl, split_file=None,
svnuser=None, svnpasswd=None,
pollinterval=10*60, histmax=100,
- svnbin='svn'):
+ svnbin='svn', schedulerNames=None):
"""
@type svnurl: string
@param svnurl: the SVN URL that describes the repository and
@@ -163,6 +163,12 @@
@param svnbin: path to svn binary, defaults to just 'svn'. Use
this if your subversion command lives in an
unusual location.
+
+ @type schedulerNames: list or None
+ @param schedulerNames: a list of Scheduler names. When a change is
+ received, it will be handed to the list of
+ schedulers to decide the outcome.
+
"""
if svnurl.endswith("/"):
@@ -171,6 +177,14 @@
self.split_file_function = split_file or split_file_alwaystrunk
self.svnuser = svnuser
self.svnpasswd = svnpasswd
+ errmsg = ("The schedulerNames= argument to Pollers must be a list "
+ "of Scheduler description names (i.e. the 'name' key of the "
+ "Scheduler specification tuple)")
+ if schedulerNames:
+ assert isinstance(schedulerNames, (list, tuple)), errmsg
+ for s in schedulerNames:
+ assert isinstance(s, str), errmsg
+ self.schedulerNames = schedulerNames
self.svnbin = svnbin
self.pollinterval = pollinterval
@@ -431,7 +445,7 @@
def submit_changes(self, changes):
for c in changes:
- self.parent.addChange(c)
+ self.parent.addChange(c, self.schedulerNames)
def finished_ok(self, res):
log.msg("SVNPoller finished polling")
diff -Nuar buildbot-0.7.7/buildbot/master.py buildbot-0.7.7-scheduler/buildbot/master.py
--- buildbot-0.7.7/buildbot/master.py 2008-03-29 19:30:17.000000000 -0700
+++ buildbot-0.7.7-scheduler/buildbot/master.py 2008-04-30 09:43:30.000000000 -0700
@@ -854,9 +854,13 @@
return d
- def addChange(self, change):
+ def addChange(self, change, schedulerNames):
for s in self.allSchedulers():
- s.addChange(change)
+ if schedulerNames:
+ if s.name in schedulerNames:
+ s.addChange(change)
+ else:
+ s.addChange(change)
def submitBuildSet(self, bs):
# determine the set of Builders to use
Thanks,
Minesh
More information about the devel
mailing list