[Buildbot-devel] one master - multiple projects
Mark Pauley
mpauley at apple.com
Fri May 12 18:47:18 UTC 2006
I'm not sure. I actually don't mind the unimportant changes coming
through like that, because it's visual feedback as to what's going on
with the repository.
As for the regex stuff, I think that only compiling the regex takes a
lot of time. My new code looks like:
def isChangeImportant(project_name_expr, change):
for file in change.files:
#log.msg("change: " + file + "\n")
if project_name_expr.match(file) != None:
return True
return False
def make_importanceTest(project_name):
project_name_expr = re.compile(project_name)
return lambda change: isChangeImportant(project_name_expr,
change)
As you can see, there is an upfront cost to creating the regex for
each of the schedulers, but then it gets cached into the closure.
I realized that before I was creating a new re for each importance
test (which requires creating an FSA and minimization of the FSA etc.)
This has made a big difference for my buildbot master's performance,
while still allowing me to test for important changes in a very
flexible manner.
thanks
_Mark
On May 12, 2006, at 11:29 AM, Alexander Lorenz wrote:
> hi mark,
>
> so now i finally found some time to spend on your code.
>
> thanks again!
>
>
> great little trick with the subclassed scheduler there.
>
>
>
> but all the other stuff you can have shorter, simpler and probably
> faster, no regexp etc. (regexps are pretty slow i gather):
>
>
> in the master.cfg, just take this method:
>
> def determineImportantChange( importantNamesList ):
> def checkChangeList( thisChange ):
> for importantName in importantNamesList:
> for filename in thisChange.files:
> if filename.find( importantName ) != -1:
> return True
> return False
> return checkChangeList
>
>
> and use it in your schedulers like this:
>
> c['schedulers'].append( OnlyImportantScheduler
> ( name="fooProjectScheduler",
>
> treeStableTimer=30*60,
>
> branch=None,
>
> builderNames=["fooProjectBuilder_SuSe", "fooProjectBuilder_Win"],
>
> fileIsImportant=determineImportantChange( ["fooMainProject",
> "fooSubProject"] ) ) )
>
>
>
> if i'm not mistaken, you don't need to set/calculate the project
> name like you do, it should be part of the filename in the change
> object which stores the whole absolute path of the changed file.
> so if your cvs module name is your project's name /(which should
> normally be the case), then it's name is already in the filename
> string in the change object.
>
> this works fine for me.
>
> the scheduler listens for changes that are part of any project
> which's name you pass in the parameter list for
> determineImportantChange.
> so if one project build should also be triggered by check-ins to
> another projects which it somehow depends on, no problem.
> i use this a lot in our buildbot, since we have a number of project
> interdependencies.
>
>
> one problems remains though:
>
> the scheduler hack fixes the problem with improper blamelists.
>
> but the unimportant changes still appear on the watefall display,
> which is still pretty unnerving.
>
> can you somehow tell the waterfall status to only use important
> changes too?
>
>
>
>
>
> regards
>
> alex
>
>
>
>
>
>
> Alexander Lorenz wrote:
>> thanks for your sources!
>>
>> i'll try to hack them in as soon as possible. ;)
>>
>>
>>
>> regards
>>
>> alex
>>
>>
>>
>> Mark Pauley wrote:
>>> I've accomplished just this.
>>>
>>> First, I hacked the svn_buildbot.py script to report the
>>> repository name in the changes
>>>
>>> fooProj/trunk/foo.txt
>>>
>>> instead of trunk/foo.txt
>>>
>>> then, I subclassed the Scheduler class like so:
>>>
>>> RCS file: /cvsroot/buildbot/buildbot/buildbot/scheduler.py,v
>>> retrieving revision 1.15
>>> diff -U5 -r1.15 scheduler.py
>>> --- buildbot/scheduler.py 12 Mar 2006 11:28:04 -0000 1.15
>>> +++ buildbot/scheduler.py 4 May 2006 17:34:08 -0000
>>> @@ -180,10 +180,13 @@
>>>
>>> def stopService(self):
>>> self.stopTimer()
>>> return service.MultiService.stopService(self)
>>>
>>> +class OnlyImportantScheduler(Scheduler):
>>> + def addUnimportantChange(self, change):
>>> + log.msg("%s: change is not important, discarding %s" %
>>> (self, change))
>>>
>>> class AnyBranchScheduler(BaseUpstreamScheduler):
>>> """This Scheduler will handle changes on a variety of
>>> branches. It will
>>> accumulate Changes for each branch separately. It works by
>>> creating a
>>> separate Scheduler for each new branch it sees."""
>>>
>>>
>>> and lastly, I did the following to my master.cfg:
>>>
>>> thisMaster_projects = [
>>> "foo",
>>> "bar",
>>> "baz"
>>> ]
>>>
>>> ...
>>>
>>> def isChangeImportant(project_name, change):
>>> project_name_expr = re.compile(project_name)
>>>
>>> for file in change.files:
>>> if project_name_expr.match(file) != None:
>>> return True
>>>
>>> return False
>>>
>>>
>>> def make_importanceTest(project_name):
>>> return lambda change: isChangeImportant(project_name,
>>> change)
>>>
>>>
>>>
>>> #fix this to get it's list from the same place as the builders
>>> for project in thisMaster_projects:
>>> c['schedulers'].append(OnlyImportantScheduler(name=project
>>> + "_thisMaster",
>>> branch=None,
>>> treeStableTimer=2*60,
>>> builderNames=[project +
>>> "_thisMaster"],
>>> fileIsImportant=
>>> (make_importanceTest(project + "/trunk"))
>>> ) )
>>>
>>> then set up the builders for each of those projects with the same
>>> name "foo_thisMaster", etc.
>>> and you should be good.
>>>
>>> The krux was OnlyImportantScheduler and the closure-based trick
>>> for generating importance tests on the fly.
>>> Not re.compiling every time is also possible by moving the regular
>>> expression object you're matching against
>>> into the closure.
>>>
>>>
>>> _Mark
>>>
>>>
>>>
>>> On May 4, 2006, at 7:44 AM, Alexander Lorenz wrote:
>>>
>>>> hello,
>>>>
>>>> as discussed in former mailings, it is obvious that buildbot was
>>>> originally designed to be used in a way that assigns one project
>>>> to one master.
>>>> as also discussed, this is a limitation that does not meet
>>>> everybody's building/testing requirements,and we are one of these
>>>> companies.
>>>> to avoid redundancies and for the sake of simplicity of use for
>>>> all involved developers, it is very desirable for us to have one
>>>> master handle all our projects.
>>>>
>>>> when you try to do this, several obvious issues occur, and i
>>>> don't know how far these have been noticed and made part of
>>>> future plans for buildbot.
>>>> the so far most important question being: is there a way that
>>>> only those changes are shown in the waterfall display, the mails,
>>>> the blamelist and so on, that are important for one of the
>>>> schedulers, and not just _all_ changes?
>>>> we have a very large cvs, lots of people committing, but only a
>>>> few projects built with buildbot.
>>>> the current situation leads to a bloated up waterfall display
>>>> with changes that noone is interested in, an incorrect blamelist
>>>> and change list with authors that commited _some_thing, bu
>>>> nothing of importance to the project.
>>>>
>>>>
>>>> thanks and regards
>>>>
>>>> alex
>>>>
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------
>>>> Using Tomcat but need to do more? Need to support web services,
>>>> security?
>>>> Get stuff done quickly with pre-integrated technology to make
>>>> your job easier
>>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache
>>>> Geronimo
>>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>>> _______________________________________________
>>>> Buildbot-devel mailing list
>>>> Buildbot-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>
>>>
>>
>>
>>
>> -------------------------------------------------------
>> Using Tomcat but need to do more? Need to support web services,
>> security?
>> Get stuff done quickly with pre-integrated technology to make your
>> job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache
>> Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> Buildbot-devel mailing list
>> Buildbot-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>
>
More information about the devel
mailing list