[Buildbot-devel] one master - multiple projects

Mark Pauley mpauley at apple.com
Fri May 12 18:47:18 UTC 2006


I'm not sure.  I actually don't mind the unimportant changes coming  
through like that, because it's visual feedback as to what's going on  
with the repository.

As for the regex stuff, I think that only compiling the regex takes a  
lot of time.  My new code looks like:

def isChangeImportant(project_name_expr, change):
         for file in change.files:
                 #log.msg("change: " + file + "\n")
                 if project_name_expr.match(file) != None:
                         return True

         return False


def make_importanceTest(project_name):
         project_name_expr = re.compile(project_name)
         return lambda change: isChangeImportant(project_name_expr,  
change)


As you can see, there is an upfront cost to creating the regex for  
each of the schedulers, but then it gets cached into the closure.
I realized that before I was creating a new re for each importance  
test (which requires creating an FSA and minimization of the FSA etc.)

This has made a big difference for my buildbot master's performance,  
while still allowing me to test for important changes in a very  
flexible manner.

thanks

_Mark


On May 12, 2006, at 11:29 AM, Alexander Lorenz wrote:

> hi mark,
>
> so now i finally found some time to spend on your code.
>
> thanks again!
>
>
> great little trick with the subclassed scheduler there.
>
>
>
> but all the other stuff you can have shorter, simpler and probably  
> faster, no regexp etc. (regexps are pretty slow i gather):
>
>
> in the master.cfg, just take this method:
>
> def determineImportantChange( importantNamesList ):
>    def checkChangeList( thisChange ):
>         for importantName in importantNamesList:
>             for filename in thisChange.files:
>                 if filename.find( importantName ) != -1:
>                     return True
>         return False
>    return checkChangeList
>
>
> and use it in your schedulers like this:
>
> c['schedulers'].append( OnlyImportantScheduler 
> ( name="fooProjectScheduler",
>                                                                
> treeStableTimer=30*60,
>                                                                
> branch=None,
>                                                                
> builderNames=["fooProjectBuilder_SuSe", "fooProjectBuilder_Win"],
>                                                                
> fileIsImportant=determineImportantChange( ["fooMainProject",  
> "fooSubProject"] ) ) )
>
>
>
> if i'm not mistaken, you don't need to set/calculate the project  
> name like you do, it should be part of the filename in the change  
> object which stores the whole absolute path of the changed file.
> so if your cvs module name is your project's name /(which should  
> normally be the case), then it's name is already in the filename  
> string in the change object.
>
> this works fine for me.
>
> the scheduler listens for changes that are part of any project  
> which's name you pass in the parameter list for  
> determineImportantChange.
> so if one project build should also be triggered by check-ins to  
> another projects which it somehow depends on, no problem.
> i use this a lot in our buildbot, since we have a number of project  
> interdependencies.
>
>
> one problems remains though:
>
> the scheduler hack fixes the problem with improper blamelists.
>
> but the unimportant changes still appear on the watefall display,  
> which is still pretty unnerving.
>
> can you somehow tell the waterfall status to only use important  
> changes too?
>
>
>
>
>
> regards
>
> alex
>
>
>
>
>
>
> Alexander Lorenz wrote:
>> thanks for your sources!
>>
>> i'll try to hack them in as soon as possible. ;)
>>
>>
>>
>> regards
>>
>> alex
>>
>>
>>
>> Mark Pauley wrote:
>>> I've accomplished just this.
>>>
>>> First, I hacked the svn_buildbot.py script to report the  
>>> repository name in the changes
>>>
>>> fooProj/trunk/foo.txt
>>>
>>> instead of trunk/foo.txt
>>>
>>> then, I subclassed the Scheduler class like so:
>>>
>>> RCS file: /cvsroot/buildbot/buildbot/buildbot/scheduler.py,v
>>> retrieving revision 1.15
>>> diff -U5 -r1.15 scheduler.py
>>> --- buildbot/scheduler.py       12 Mar 2006 11:28:04 -0000      1.15
>>> +++ buildbot/scheduler.py       4 May 2006 17:34:08 -0000
>>> @@ -180,10 +180,13 @@
>>>
>>>      def stopService(self):
>>>          self.stopTimer()
>>>          return service.MultiService.stopService(self)
>>>
>>> +class OnlyImportantScheduler(Scheduler):
>>> +    def addUnimportantChange(self, change):
>>> +        log.msg("%s: change is not important, discarding %s" %  
>>> (self, change))
>>>
>>>  class AnyBranchScheduler(BaseUpstreamScheduler):
>>>      """This Scheduler will handle changes on a variety of  
>>> branches. It will
>>>      accumulate Changes for each branch separately. It works by  
>>> creating a
>>>      separate Scheduler for each new branch it sees."""
>>>
>>>
>>> and lastly, I did the following to my master.cfg:
>>>
>>> thisMaster_projects = [
>>>                       "foo",
>>>                       "bar",
>>>                       "baz"
>>>                     ]
>>>
>>> ...
>>>
>>> def isChangeImportant(project_name, change):
>>>         project_name_expr = re.compile(project_name)
>>>
>>>         for file in change.files:
>>>                 if project_name_expr.match(file) != None:
>>>                         return True
>>>
>>>         return False
>>>
>>>
>>> def make_importanceTest(project_name):
>>>         return lambda change: isChangeImportant(project_name,  
>>> change)
>>>
>>>
>>>
>>> #fix this to get it's list from the same place as the builders
>>> for project in thisMaster_projects:
>>>         c['schedulers'].append(OnlyImportantScheduler(name=project  
>>> + "_thisMaster",
>>>                                          branch=None,
>>>                                          treeStableTimer=2*60,
>>>                                          builderNames=[project +  
>>> "_thisMaster"],
>>>                                          fileIsImportant= 
>>> (make_importanceTest(project + "/trunk"))
>>>                                 ) )
>>>
>>> then set up the builders for each of those projects with the same  
>>> name "foo_thisMaster",  etc.
>>> and you should be good.
>>>
>>> The krux was OnlyImportantScheduler and the closure-based trick  
>>> for generating importance tests on the fly.
>>> Not re.compiling every time is also possible by moving the regular  
>>> expression object you're matching against
>>> into the closure.
>>>
>>>
>>> _Mark
>>>
>>>
>>>
>>> On May 4, 2006, at 7:44 AM, Alexander Lorenz wrote:
>>>
>>>> hello,
>>>>
>>>> as discussed in former mailings, it is obvious that buildbot was  
>>>> originally designed to be used in a way that assigns one project  
>>>> to one master.
>>>> as also discussed, this is a limitation that does not meet  
>>>> everybody's building/testing requirements,and we are one of these  
>>>> companies.
>>>> to avoid redundancies and for the sake of simplicity of use for  
>>>> all involved developers, it is very desirable for us to have one  
>>>> master handle  all our projects.
>>>>
>>>> when you try to do this, several obvious issues occur, and i  
>>>> don't know how far these have been noticed and made part of  
>>>> future plans for buildbot.
>>>> the so far most important question being: is there a way that  
>>>> only those changes are shown in the waterfall display, the mails,  
>>>> the blamelist and so on, that are important for one of the  
>>>> schedulers, and not just _all_ changes?
>>>> we have a very large cvs, lots of people committing, but only a  
>>>> few projects built with buildbot.
>>>> the current situation leads to a bloated up waterfall display  
>>>> with changes that noone is interested in, an incorrect blamelist  
>>>> and change list with authors that commited _some_thing, bu  
>>>> nothing of importance to the project.
>>>>
>>>>
>>>> thanks and regards
>>>>
>>>> alex
>>>>
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------
>>>> Using Tomcat but need to do more? Need to support web services,  
>>>> security?
>>>> Get stuff done quickly with pre-integrated technology to make  
>>>> your job easier
>>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache  
>>>> Geronimo
>>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>>> _______________________________________________
>>>> Buildbot-devel mailing list
>>>> Buildbot-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>
>>>
>>
>>
>>
>> -------------------------------------------------------
>> Using Tomcat but need to do more? Need to support web services,  
>> security?
>> Get stuff done quickly with pre-integrated technology to make your  
>> job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache  
>> Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> Buildbot-devel mailing list
>> Buildbot-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>
>





More information about the devel mailing list