[Buildbot-devel] Multiple change sources and scheduler issues

Minesh Patel mipatel at vmware.com
Tue Jul 1 17:33:33 UTC 2008

Message: 6
Date: Tue, 1 Jul 2008 10:28:06 -0400
From: "Dustin J. Mitchell" <dustin at zmanda.com>
Subject: Re: [Buildbot-devel] Multiple change sources and scheduler
To: "Duncan Ferguson" <duncan.ferguson at altinity.com>
Cc: buildbot-devel at lists.sourceforge.net
        <42338fbf0807010728j5c46a3e4i40cd902c9f4916b6 at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

On Tue, Jul 1, 2008 at 8:34 AM, Duncan Ferguson
<duncan.ferguson at altinity.com> wrote:
> I have set up my buildbot v0.77 with a number of change sources and
> Schedulers.  The problem I have is that when a change is detected all
> schedulers are triggered, not just the one configured for the
> particular svn repository.

This is a shortcoming of the current buildbot architecture.  The only
workaround I know is to run multiple buildmaster processes.  That's
what we do.  I've been toying with the idea of running multiple
BuildMaster services inside a single application (by futzing with the
buildbot.tac), but that's a pretty awful hack, and still requires each
slave machine to run a buildslave instance for each buildmaster.

We're about to add a half-dozen additional buildmasters, and each
buildmaster process already has a virtual size of 0.5G, so I'm a
little worried about this issue, too :)


Here is a patch for _ONLY_ svn that I have been using successfully. Just how builders are identified in the schedulers, I added a list of how schedulers are identified in the change_sources. Let me know if there are better ideas, I don't mind updating it and submitting a patch but it seems like there would have to be changes to all the 'pollers'. Let me know what you think. Sorry there are no tests yet, but can be added once we figure out if this is the approach we want to take.

In the master.cfg
The schedulerNames would have to be declared on
c['change_source'] = [SVNPoller(svnurl=url,
c['schedulers'] = [Scheduler(name='foo',

Here is the patch:
diff -Nuar buildbot-0.7.7/buildbot/changes/changes.py buildbot-0.7.7-scheduler/buildbot/changes/changes.py
--- buildbot-0.7.7/buildbot/changes/changes.py  2008-03-29 19:30:16.000000000 -0700
+++ buildbot-0.7.7-scheduler/buildbot/changes/changes.py        2008-04-30 09:35:08.000000000 -0700
@@ -205,17 +205,17 @@
         d = defer.maybeDeferred(source.disownServiceParent)
         return d

-    def addChange(self, change):
+    def addChange(self, change, schedulerNames):
         """Deliver a file change event. The event should be a Change object.
         This method will timestamp the object as it is received."""
         log.msg("adding change, who %s, %d files, rev=%s, branch=%s, "
-                "comments %s" % (change.who, len(change.files),
+                "comments %s to schedulers %r" % (change.who, len(change.files),
                                  change.revision, change.branch,
-                                 change.comments))
+                                 change.comments, schedulerNames))
         change.number = self.nextNumber
         self.nextNumber += 1
-        self.parent.addChange(change)
+        self.parent.addChange(change, schedulerNames)
         # TODO: call pruneChanges after a while

     def pruneChanges(self):
diff -Nuar buildbot-0.7.7/buildbot/changes/svnpoller.py buildbot-0.7.7-scheduler/buildbot/changes/svnpoller.py
--- buildbot-0.7.7/buildbot/changes/svnpoller.py        2008-03-29 19:30:17.000000000 -0700
+++ buildbot-0.7.7-scheduler/buildbot/changes/svnpoller.py      2008-04-30 09:42:28.000000000 -0700
@@ -47,7 +47,7 @@
     compare_attrs = ["svnurl", "split_file_function",
                      "svnuser", "svnpasswd",
                      "pollinterval", "histmax",
-                     "svnbin"]
+                     "svnbin", "schedulerNames"]

     parent = None # filled in when we're added
     last_change = None
@@ -57,7 +57,7 @@
     def __init__(self, svnurl, split_file=None,
                  svnuser=None, svnpasswd=None,
                  pollinterval=10*60, histmax=100,
-                 svnbin='svn'):
+                 svnbin='svn', schedulerNames=None):
         @type  svnurl: string
         @param svnurl: the SVN URL that describes the repository and
@@ -163,6 +163,12 @@
         @param svnbin:       path to svn binary, defaults to just 'svn'. Use
                              this if your subversion command lives in an
                              unusual location.
+        @type  schedulerNames: list or None
+        @param schedulerNames: a list of Scheduler names. When a change is
+                               received, it will be handed to the list of
+                               schedulers to decide the outcome.

         if svnurl.endswith("/"):
@@ -171,6 +177,14 @@
         self.split_file_function = split_file or split_file_alwaystrunk
         self.svnuser = svnuser
         self.svnpasswd = svnpasswd
+        errmsg = ("The schedulerNames= argument to Pollers must be a list "
+                  "of Scheduler description names (i.e. the 'name' key of the "
+                  "Scheduler specification tuple)")
+        if schedulerNames:
+            assert isinstance(schedulerNames, (list, tuple)), errmsg
+            for s in schedulerNames:
+                assert isinstance(s, str), errmsg
+        self.schedulerNames = schedulerNames

         self.svnbin = svnbin
         self.pollinterval = pollinterval
@@ -431,7 +445,7 @@

     def submit_changes(self, changes):
         for c in changes:
-            self.parent.addChange(c)
+            self.parent.addChange(c, self.schedulerNames)

     def finished_ok(self, res):
         log.msg("SVNPoller finished polling")
diff -Nuar buildbot-0.7.7/buildbot/master.py buildbot-0.7.7-scheduler/buildbot/master.py
--- buildbot-0.7.7/buildbot/master.py   2008-03-29 19:30:17.000000000 -0700
+++ buildbot-0.7.7-scheduler/buildbot/master.py 2008-04-30 09:43:30.000000000 -0700
@@ -854,9 +854,13 @@
         return d

-    def addChange(self, change):
+    def addChange(self, change, schedulerNames):
         for s in self.allSchedulers():
-            s.addChange(change)
+            if schedulerNames:
+                if s.name in schedulerNames:
+                    s.addChange(change)
+            else:
+                s.addChange(change)

     def submitBuildSet(self, bs):
         # determine the set of Builders to use


More information about the devel mailing list