[Buildbot-devel] one master - multiple projects

Fri May 12 18:29:07 UTC 2006

hi mark,

so now i finally found some time to spend on your code.

thanks again!

great little trick with the subclassed scheduler there.

but all the other stuff you can have shorter, simpler and probably 
faster, no regexp etc. (regexps are pretty slow i gather):

in the master.cfg, just take this method:

def determineImportantChange( importantNamesList ):
    def checkChangeList( thisChange ):
         for importantName in importantNamesList:
             for filename in thisChange.files:
                 if filename.find( importantName ) != -1:
                     return True
         return False
    return checkChangeList

and use it in your schedulers like this:

c['schedulers'].append( OnlyImportantScheduler( name="fooProjectScheduler",

treeStableTimer=30*60,
                                                               branch=None,

builderNames=["fooProjectBuilder_SuSe", "fooProjectBuilder_Win"],

fileIsImportant=determineImportantChange( ["fooMainProject", 
"fooSubProject"] ) ) )

if i'm not mistaken, you don't need to set/calculate the project name 
like you do, it should be part of the filename in the change object 
which stores the whole absolute path of the changed file.
so if your cvs module name is your project's name /(which should 
normally be the case), then it's name is already in the filename string 
in the change object.

this works fine for me.

the scheduler listens for changes that are part of any project which's 
name you pass in the parameter list for determineImportantChange.
so if one project build should also be triggered by check-ins to another 
projects which it somehow depends on, no problem.
i use this a lot in our buildbot, since we have a number of project 
interdependencies.

one problems remains though:

the scheduler hack fixes the problem with improper blamelists.

but the unimportant changes still appear on the watefall display, which 
is still pretty unnerving.

can you somehow tell the waterfall status to only use important changes too?

regards

alex

Alexander Lorenz wrote:
> thanks for your sources!
>
> i'll try to hack them in as soon as possible. ;)
>
>
>
> regards
>
> alex
>
>
>
> Mark Pauley wrote:
>> I've accomplished just this.
>>
>> First, I hacked the svn_buildbot.py script to report the repository 
>> name in the changes
>>
>> fooProj/trunk/foo.txt
>>
>> instead of trunk/foo.txt
>>
>> then, I subclassed the Scheduler class like so:
>>
>> RCS file: /cvsroot/buildbot/buildbot/buildbot/scheduler.py,v
>> retrieving revision 1.15
>> diff -U5 -r1.15 scheduler.py
>> --- buildbot/scheduler.py       12 Mar 2006 11:28:04 -0000      1.15
>> +++ buildbot/scheduler.py       4 May 2006 17:34:08 -0000
>> @@ -180,10 +180,13 @@
>>
>>      def stopService(self):
>>          self.stopTimer()
>>          return service.MultiService.stopService(self)
>>
>> +class OnlyImportantScheduler(Scheduler):
>> +    def addUnimportantChange(self, change):
>> +        log.msg("%s: change is not important, discarding %s" % 
>> (self, change))
>>
>>  class AnyBranchScheduler(BaseUpstreamScheduler):
>>      """This Scheduler will handle changes on a variety of branches. 
>> It will
>>      accumulate Changes for each branch separately. It works by 
>> creating a
>>      separate Scheduler for each new branch it sees."""
>>
>>
>> and lastly, I did the following to my master.cfg:
>>
>> thisMaster_projects = [
>>                       "foo",
>>                       "bar",
>>                       "baz"
>>                     ]
>>
>> ...
>>
>> def isChangeImportant(project_name, change):
>>         project_name_expr = re.compile(project_name)
>>
>>         for file in change.files:
>>                 if project_name_expr.match(file) != None:
>>                         return True
>>
>>         return False
>>
>>
>> def make_importanceTest(project_name):
>>         return lambda change: isChangeImportant(project_name, change)
>>
>>
>>
>> #fix this to get it's list from the same place as the builders
>> for project in thisMaster_projects:
>>         c['schedulers'].append(OnlyImportantScheduler(name=project + 
>> "_thisMaster",
>>                                          branch=None,
>>                                          treeStableTimer=2*60,
>>                                          builderNames=[project + 
>> "_thisMaster"],
>>                                          
>> fileIsImportant=(make_importanceTest(project + "/trunk"))
>>                                 ) )
>>
>> then set up the builders for each of those projects with the same 
>> name "foo_thisMaster",  etc.
>> and you should be good.
>>
>> The krux was OnlyImportantScheduler and the closure-based trick for 
>> generating importance tests on the fly.
>> Not re.compiling every time is also possible by moving the regular 
>> expression object you're matching against
>> into the closure.
>>
>>
>> _Mark
>>
>>
>>
>> On May 4, 2006, at 7:44 AM, Alexander Lorenz wrote:
>>
>>> hello,
>>>
>>> as discussed in former mailings, it is obvious that buildbot was 
>>> originally designed to be used in a way that assigns one project to 
>>> one master.
>>> as also discussed, this is a limitation that does not meet 
>>> everybody's building/testing requirements,and we are one of these 
>>> companies.
>>> to avoid redundancies and for the sake of simplicity of use for all 
>>> involved developers, it is very desirable for us to have one master 
>>> handle  all our projects.
>>>
>>> when you try to do this, several obvious issues occur, and i don't 
>>> know how far these have been noticed and made part of future plans 
>>> for buildbot.
>>> the so far most important question being: is there a way that only 
>>> those changes are shown in the waterfall display, the mails, the 
>>> blamelist and so on, that are important for one of the schedulers, 
>>> and not just _all_ changes?
>>> we have a very large cvs, lots of people committing, but only a few 
>>> projects built with buildbot.
>>> the current situation leads to a bloated up waterfall display with 
>>> changes that noone is interested in, an incorrect blamelist and 
>>> change list with authors that commited _some_thing, bu nothing of 
>>> importance to the project.
>>>
>>>
>>> thanks and regards
>>>
>>> alex
>>>
>>>
>>>
>>>
>>> -------------------------------------------------------
>>> Using Tomcat but need to do more? Need to support web services, 
>>> security?
>>> Get stuff done quickly with pre-integrated technology to make your 
>>> job easier
>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache 
>>> Geronimo
>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 
>>>
>>> _______________________________________________
>>> Buildbot-devel mailing list
>>> Buildbot-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>
>>
>
>
>
> -------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job 
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache 
> Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>