[Buildbot-devel] Multiple upstreams for Dependent scheduler

[Dan Kegel]

Filed http://trac.buildbot.net/ticket/2917 as a cautionary tale about
the Dependent scheduler.  Don't know if it's still relevant.
- Dan

On Thu, Sep 25, 2014 at 1:14 PM, Dan Kegel <dank at kegel.com> wrote:
> OK, I'll bite, can you post your MultiDependent scheduler?
>
> (excruciating detail:
>
> I need to implement some form of dependencies between builders on our buildbot.
>
> Our buildbot has C++ projects which subclass each other, like this:
>   libBase
>   libDerived (subclasses libBase)
>   app (instantiates objects of classes Base and Derived)
> Each of these are in their own git repo and have their own builder
> with a SingleBranchScheduler and a ForceScheduler.
>
> When somebody adds a method to libBase, both libDerived and app need
> to be rebuilt.
> Currently, we do this by clicking Force on both, but it's easy to
> forget one (the actual
> situation involves four or so libraries and six or so apps), which has
> caused headaches.
>
> Buildbot provides Triggerable schedulers, so in principle I could use
> one of those to
> trigger builds of libDerived and app when libBase changes, etc.
>
> However, there is a complication.  If the builds are triggered in the
> wrong order (app first
> and then libDerived), app will fail its unit tests.   For instance, if
> lots of slaves are available,
> even if the builds are triggered in the right order, they will run in
> parallel, and app's tests will
> fail because it's running an out of date libDerived.
> I could prevent that by using waitForFinish=true, but I'm afraid I'll
> run out of buildslaves quickly
> if I do that (all my buildslaves are exclusive and reboot after each
> build), and deadlock.
>
> I have a funny feeling more machinery inside buildbot is needed to make this
> scenario work.  How about if the Dependent scheduler allowed multiple upstreams,
> and triggered the build only when the sum of pending + active builds
> transitioned
> from > 0 to 0, and all were green?  That would probably skip
> intermediate builds if
> commits come fast and furious on the upstream projects, but maybe
> that's a good thing.)
>
>
> On Fri, Dec 7, 2012 at 9:06 AM, Andy Salnikov <a_salnikov at yahoo.com> wrote:
>> Hi Charles,
>>
>> thanks for pointing out Triggerable option. I never used that before and
>> needed tolearn how it works. My impression that it should work well
>> even though configuration may become a bit more complex.
>>
>> Meantime I wrote simple MultiDependent scheduler for myself, it works
>> but it's not very trivial. If anybody is interested I could post it here.
>>
>> Cheers,
>> Andy
>>
>> On 12/4/2012 5:23 PM, Charles Lepple wrote:
>>> On Dec 4, 2012, at 2:06 PM, Andy Salnikov wrote:
>>>
>>>> I'm trying to setup new build system with buildbot and one thing that I'd
>>>> like to do for it is to run some action when "all" builders have finished.
>>>> Dependent scheduler seems to the kind that I need, but its limitation
>>>> is that it only depends on a single upstream scheduler while I want to
>>>> wait for multiple schedulers (running in parallel) to finish. Would it be
>>>> possible to extend the Dependent scheduler to support multiple upstream
>>>> schedules (with an "AND" logic)? I gather this could be useful for other
>>>> people as well, I of course could (and maybe would) implement my own
>>>> scheduler but having this in standard code base could be useful.
>>> Can you make the upstream schedulers into Triggerable schedulers, and have another builder trigger those schedulers synchronously? Then, after the Trigger step, you perform the action that depends on all of the other actions.
>>>
>>> Scheduler:
>>>
>>> http://buildbot.net/buildbot/docs/current/manual/cfg-schedulers.html#triggerable-scheduler
>>>
>>> Trigger step:
>>>
>>> http://buildbot.net/buildbot/docs/current/manual/cfg-buildsteps.html#triggering-schedulers
>>>
>>>> Another thing that I'd like to have in Dependent scheduler is "forwarding"
>>>> of the properties (all or selected subset of them) from the upstream
>>>> scheduler. I have a special nightly scheduler which defines some special
>>>> properties used throughout the system. With that I had also subclass a
>>>> Dependent scheduler to pass properties defined in the nightly scheduler
>>>> down to the builders. Simpler solution that does not need subclassing
>>>> would be preferable to me.
>>> I haven't used the set_properties option with Interpolate, but the older copy_properties worked for me.
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
>> Remotely access PCs and mobile devices and provide instant support
>> Improve your efficiency, and focus on delivering more value-add services
>> Discover what IT Professionals Know. Rescue delivers
>> http://p.sf.net/sfu/logmein_12329d2d
>> _______________________________________________
>> Buildbot-devel mailing list
>> Buildbot-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel