[Buildbot-devel] Multiple upstreams for Dependent scheduler

[Dan Kegel]

OK, I'll bite, can you post your MultiDependent scheduler?

(excruciating detail:

I need to implement some form of dependencies between builders on our buildbot.

Our buildbot has C++ projects which subclass each other, like this:
  libBase
  libDerived (subclasses libBase)
  app (instantiates objects of classes Base and Derived)
Each of these are in their own git repo and have their own builder
with a SingleBranchScheduler and a ForceScheduler.

When somebody adds a method to libBase, both libDerived and app need
to be rebuilt.
Currently, we do this by clicking Force on both, but it's easy to
forget one (the actual
situation involves four or so libraries and six or so apps), which has
caused headaches.

Buildbot provides Triggerable schedulers, so in principle I could use
one of those to
trigger builds of libDerived and app when libBase changes, etc.

However, there is a complication.  If the builds are triggered in the
wrong order (app first
and then libDerived), app will fail its unit tests.   For instance, if
lots of slaves are available,
even if the builds are triggered in the right order, they will run in
parallel, and app's tests will
fail because it's running an out of date libDerived.
I could prevent that by using waitForFinish=true, but I'm afraid I'll
run out of buildslaves quickly
if I do that (all my buildslaves are exclusive and reboot after each
build), and deadlock.

I have a funny feeling more machinery inside buildbot is needed to make this
scenario work.  How about if the Dependent scheduler allowed multiple upstreams,
and triggered the build only when the sum of pending + active builds
transitioned
from > 0 to 0, and all were green?  That would probably skip
intermediate builds if
commits come fast and furious on the upstream projects, but maybe
that's a good thing.)


On Fri, Dec 7, 2012 at 9:06 AM, Andy Salnikov <a_salnikov at yahoo.com> wrote:
> Hi Charles,
>
> thanks for pointing out Triggerable option. I never used that before and
> needed tolearn how it works. My impression that it should work well
> even though configuration may become a bit more complex.
>
> Meantime I wrote simple MultiDependent scheduler for myself, it works
> but it's not very trivial. If anybody is interested I could post it here.
>
> Cheers,
> Andy
>
> On 12/4/2012 5:23 PM, Charles Lepple wrote:
>> On Dec 4, 2012, at 2:06 PM, Andy Salnikov wrote:
>>
>>> I'm trying to setup new build system with buildbot and one thing that I'd
>>> like to do for it is to run some action when "all" builders have finished.
>>> Dependent scheduler seems to the kind that I need, but its limitation
>>> is that it only depends on a single upstream scheduler while I want to
>>> wait for multiple schedulers (running in parallel) to finish. Would it be
>>> possible to extend the Dependent scheduler to support multiple upstream
>>> schedules (with an "AND" logic)? I gather this could be useful for other
>>> people as well, I of course could (and maybe would) implement my own
>>> scheduler but having this in standard code base could be useful.
>> Can you make the upstream schedulers into Triggerable schedulers, and have another builder trigger those schedulers synchronously? Then, after the Trigger step, you perform the action that depends on all of the other actions.
>>
>> Scheduler:
>>
>> http://buildbot.net/buildbot/docs/current/manual/cfg-schedulers.html#triggerable-scheduler
>>
>> Trigger step:
>>
>> http://buildbot.net/buildbot/docs/current/manual/cfg-buildsteps.html#triggering-schedulers
>>
>>> Another thing that I'd like to have in Dependent scheduler is "forwarding"
>>> of the properties (all or selected subset of them) from the upstream
>>> scheduler. I have a special nightly scheduler which defines some special
>>> properties used throughout the system. With that I had also subclass a
>>> Dependent scheduler to pass properties defined in the nightly scheduler
>>> down to the builders. Simpler solution that does not need subclassing
>>> would be preferable to me.
>> I haven't used the set_properties option with Interpolate, but the older copy_properties worked for me.
>>
>
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel