[Buildbot-devel] Multiple change sources and scheduler issues

[Axel Hecht]

2008/7/1 Dustin J. Mitchell <dustin at zmanda.com>:
> On Tue, Jul 1, 2008 at 1:33 PM, Minesh Patel <mipatel at vmware.com> wrote:
>> In the master.cfg
>> The schedulerNames would have to be declared on
>> c['change_source'] = [SVNPoller(svnurl=url,
>>                                svnuser=userName,
>>                                svnpasswd=userPassword,
>>                                schedulerNames=['foo'])]
>> c['schedulers'] = [Scheduler(name='foo',
>>                             branch=None,
>>                             treeStableTimer=2*60,
>>                             builderNames=[bar])]
>
> I'm intrigued, but I'd want to defer to Brian's decision on that, as
> it's a pretty significant change.  Also, you'll have to add some
> method to ensure that current config files continue to work --
> probably just sending changes to *all* schedulers if no schedulerNames
> are specified.
>
> Brian, what do you think?

Not that I'm Brian, but I have been thinking about this some more.
We'll need something like this for our l10n builds, and I figured to
go a little bit further than just the scheduler names.

I'm going to stick to "partition the master", as I did in
http://buildbot.net/trac/ticket/301. My math/physics head thinks of
colors when you partition an area, so here's my proposal:

Add a dictionary "color" to changesources, Change, and  scheduler
(probably BaseScheduler). Here's a master.cfg snippet:

c['change_source'] = [FooSource(..., colors = dict(blue=True,
shady="pretty much"),
                                BarSource(..., colors =
dict(shady="pretty much"),
                                BazSource(...) # no colors
                              ]
c['schedulers'] = [Scheduler(name="Deep Blue", colors = dict(blue=True)),
                         Scheduler(name="Shady", colors =
dict(shady="pretty much")),
                         Scheduler(name="Gourmant") # no colors
                        ]

The change sources would pass their colors along to the Change object,
and in the scheduler entry point, it'd check the dict of the scheduler
and look if the Change matches, something like
def addColoredChange(self, change):
  if self.colors is not None:
    for color, value in self.colors.iteritems():
      if color not in change.colors or change.colors[color] != value:
        return
  self.addChange(change)

To map that to the previous example, Deep Blue would accept changes
from FooSource, Shady would accept changes from both FooSource and
BarSource, and Gourmant would eat them all.

The changes to implement this in the master and scheduler should be
rather simple, just make the master call into a new hook and implement
that on BaseScheduler, I'd need to add **kwargs to the init call
chain, probably. The changes to the ChangeSources would be rather bad,
as there's not even a general practice to have a common base class,
let alone to __init__() it with anything. I'd also had to factor out
the creation of the Change objects, but all in all, it looks like a
feasible task.

Documentation for this should probably not go into the individual
objects but rather into a new section on partitioning masters. Or
whatever we want to call it in the end.

What do you guys think?

Axel