[Buildbot-devel] custom scheduler for parallelism

[Greg MacDonald]

So getSchedulersAndProperties helps but it only schedules one and nothing else gets scheduled afterwards. This has helped decouple individual machines from the builders, which is good. But I’m having to resort to pairing schedulers and builders together one-to-one for each of the 200 commands I want to schedule. I could live with this, but I’m going to have to adjust the webpage a bit and I don’t like that there’s a hard limit to the number of slots I’ve setup. Any suggestions would be greatly appreciated. I’m using 0.8.12.

from buildbot.process.properties import Properties
from buildbot.steps.trigger import Trigger

class TriggerBuildsFromNamedParameter(Trigger):

    haltOnFailure = True
    flunkOnFailure = True

    sourceStamps = []
    alwaysUseLatest = False
    updateSourceStamp = True

    def __init__(self, maxBuilders, schedulerNames, fromParameter, toParameter, **kwargs):
        if "name" not in kwargs:
            kwargs['name'] = 'trigger'
        Trigger.__init__(self, schedulerNames=schedulerNames, **kwargs)

        self.fromParameter = fromParameter
        self.toParameter = toParameter
        self.maxBuilders = maxBuilders

    def createTriggerProperties(self, props):
        return props

    def getSchedulersAndProperties(self):
        cmds = self.build.getProperties().getProperty(self.fromParameter, [])
        if not cmds:
            raise Exception('no commands found in prop: %s' % self.fromParameter)

        if len(cmds) > self.maxBuilders:
            raise Exception('max builders reached')

        r = []
        for i, cmd in enumerate(cmds):
            schedulerName = self.schedulerNames[i]

            props = Properties()
            props.setProperty(self.toParameter, cmd, 'TriggerBuildsFromNamedParameter')
            r.append((schedulerName, props))
        return r

And the config:

def createDistributedBuilder(c, branch):
    baseName = '%sNightly' % branch.capitalize()

    factory = util.BuildFactory()
    factory.workdir = '.'

    factory.addStep(ObservedShellCommand(command=Property(navCmdPropName),
                                         description=["Export nav graphs."],
                                         timeout=3600*3,
                                         flunkOnFailure=True,
                                         haltOnFailure=True,
                                         warningPatterns=['ERROR:.*'],
                                         ))

    r = []
    for i in range(maxBuilders): # 210 in actuality
        builderName = baseName + 'NavExport_%0.4d' % i
        c['builders'].append(
            BuilderConfig(name=builderName,
              slavenames=slaveNames,
              slavebuilddir=baseName + 'NavExport',
              factory=factory,
              tags=['nightly', branch],
              ))
        r.append(builderName)

    return r

def config(c, branch, factory, coordinatorSlaveName):
    baseName = '%sNightly' % branch.capitalize()

    setupBuilderNames = createNavExportSetupBuilder(c, branch, coordinatorSlaveName)
    navExportSchedulerNames = navExportBuilderNames = createDistributedBuilder(c, branch)
    tearDownBuilderNames = createNavExportTearDownBuilder(c, branch)
    #navExportSchedulerName = baseName + 'NavExport'

    c['schedulers'].append(schedulers.Triggerable(name=baseName + 'NavExportSetup',
                                                  builderNames=setupBuilderNames))

    for builderName in navExportBuilderNames:
        c['schedulers'].append(schedulers.Triggerable(name=builderName,
                                                      builderNames=[builderName]))

    c['schedulers'].append(schedulers.Triggerable(name=baseName + 'NavExportTearDown',
                                                  builderNames=tearDownBuilderNames))

    factory.addStep(steps.SetPropertyFromCommand(command='nav_export --print_cmds',
                                                 extract_fn=lambda rc, stdout, stderr: {navCmdsPropName: loads(stdout)}))

    factory.addStep(TriggerBuildsFromNamedParameter(maxBuilders, navExportSchedulerNames, navCmdsPropName, navCmdPropName, waitForFinish=True, flunkOnFailure=True, haltOnFailure=True))

From: Greg MacDonald [mailto:gmacdonald at trionworlds.com]
Sent: Monday, July 13, 2015 11:35 AM
To: Pierre Tardy; Mikhail Sobolev; buildbot-devel at lists.sourceforge.net
Subject: Re: [Buildbot-devel] custom scheduler for parallelism

Hi Pierre,

The getSchedulersAndProperties method is just what I was looking for! Thanks a lot!

-Greg

From: Pierre Tardy [mailto:tardyp at gmail.com]
Sent: Saturday, July 11, 2015 7:45 AM
To: Mikhail Sobolev; buildbot-devel at lists.sourceforge.net<mailto:buildbot-devel at lists.sourceforge.net>
Subject: Re: [Buildbot-devel] custom scheduler for parallelism

Hi,
The Trigger step now has a getSchedulersAndProperties customization point, which can allow you to create the same build several times, but with different parameters as properties.

You can see an example in buildbot_travis:
https://github.com/tardyp/buildbot_travis/blob/master/buildbot_travis/steps/spawner.py#L51


Le sam. 11 juil. 2015 à 15:49, Mikhail Sobolev <mss at mawhrin.net<mailto:mss at mawhrin.net>> a écrit :
Hi Greg,

On Fri, Jul 10, 2015 at 08:43:02PM +0000, Greg MacDonald wrote:
>    I have a few jobs that are good parallelism candidates in that several
>    console commands can be split up between machines. I've got some basic
>    parallelism with the triggerable scheduler and multiple builders, but I
>    have to make a builder for each machine, which means I have to divvy up
>    the commands in advance. I would much prefer a more general solution where
>    I dynamically start a multiple instances of a single builder with each
>    instance having a different command to run. This way the nodes are kept
>    general and the load can be spread around the cluster dynamically. Is
>    there a way to do this that I'm not aware of? It doesn't seem like things
>    are setup this way so I was thinking of extending the triggerable
>    scheduler to do this, using the build properties to pass the command to
>    run. Does this sound like a reasonable course of action? Thx.
I wonder if you could share the snippet of your current configuration related
to your question?

--
Misha

------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Buildbot-devel mailing list
Buildbot-devel at lists.sourceforge.net<mailto:Buildbot-devel at lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/buildbot-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20150715/46dad698/attachment.html>