[users at bb.net] Trigger new build/scheduler when two related builds are finished
Yngve N. Pettersen
yngve at vivaldi.com
Mon Aug 14 09:18:09 UTC 2023
Hi again,
Nvm about the getBuild; was missing an inlineCallbacks.
However, now that I have finished registering the first trigger and started
waiting for the next one(s), the trigger step is stuck waiting for the
buildset from the idsDeferrer, which will not be available until the other
build(s) have sent their triggers.
This causes two possible issues:
1. When both/all builds are on the same (non-parallel) worker, that causes
a deadlock
2. When running on separate workers it blocks the first-posting build until
both are ready; in my production environment that means that a build that
would have finished in 2 hours, will take at least 3 hours, preventing the
next builds for that builder/worker to start.
My guess is that this cannot be resolved without a redesign of how
idsDeferred is handled by steps.Trigger.run, which is a serious
disappointment.
As mentioned earlier, the Dependent scheduler is not useful, not just
because it cannot limit the waiting to a fewer builders than all for the
upstream scheduler, but actually the Dependent scheduler is IMO *useless*
for any production environment except a single branch environment. Normal
projects will tend to have *multiple* release branches (just consider a
project with an alpha, beta, and stable branches, or a Linux distro
maintaining multiple LTS releases, and we ourselves have multiple release
branches for various product lines, normally three for the platform I am
working on for this project). This means that the Scheduler needs to be
aware of everything going on in the parent scheduler, including the
properties and sourcestamp(s) of the resulting builds.
On Sunday 13. August 2023 15:13:16 (+02:00), Yngve N. Pettersen wrote:
>
> Hello all,
>
> I have now created a variant Triggered scheduler that will wait for
multiple specific dependent builders for a given buildset.
>
> However, it looks like in base.BaseScheduler instances
self.master.db.builds.getBuild() return None for all queries, even for an
existing parent_buildid.
>
> I just completed a debug run that showed that in my special
steps.Trigger instance, correct information is available from that call,
while in the call to sch.trigger inside steps.Trigger.run() the data were
not available, inside the same call sequence.
>
> I suspect the same is the case with
self.master.db.buildrequests.getBuildRequest, used to get the buildset id,
which never gets called because the getBuild() does not return valid data.
>
> The buildset ID is necessary to run a wait for the set to complete, in
case one of the builds it is waiting for fails (or was never started).
>
> Any ideas about how to solve this issue?
>
> On Monday 31. July 2023 14:37:42 (+02:00), Yngve N. Pettersen wrote:
>
> > > Hi,
> > > Thanks for the suggestion.
> > > However, due to implementation details, specifically build version
generation, I am not sure it is practical, because there are half a dozen
other builds being triggered at the same time, by the same scheduler ( and
the build version generation and retrieval depends on the scheduler name),
and all those builds must have the same build number (although I can think
of ways to fake it).
> > > Additionally, I suspect that the information transfer used by the
universal step is not possible with that method (the data is generated by
the build jobs, and is unique for each job).
> > > Another practical issue is that this method would block the worker
where the universal step is done for at least 3 hours, preventing other
tasks it might need to do from starting until that job is complete.
> > > On Monday 31. July 2023 13:44:03 (+02:00), Roland van Laar wrote:
> > > > Hi,
> > > > To summarize, the desired workflow is:
> > > > - kick off two builds: arm64 and x64
> > > - combine output into one universal installer.
> > > > My advice use a trigger[1] with `waitForFinish`.
> > > > build = BuildFactory()
> > > build.addStep(steps.Trigger(schedulerNames=["buildx64",
"buildarm64"], waitForFinish=True)
> > > build.addStep()
> > > > Regards,
> > > > Roland
> > > > [1]
https://docs.buildbot.net/latest/manual/configuration/steps/trigger.html#trigger
> > > > > On 31-07-2023 11:00, Yngve N. Pettersen wrote:
> > > > Hello all,
> > > >
> > > > We have a build sequence that consists of two separate builds that
are then combined into a single product (specifically, this concerns Mac
x64 and arm64 builds being used to generate a universal DMG installer)
> > > >
> > > > At present we are building both sequentially on the same build
worker which makes the system work, but we are now preparing a new worker
that will be used for the arm64 build.
> > > >
> > > > The plan is to start both builds at the same time, but that
prevents the transfer of the (build properties) data necessary to combine
the products into a universal installer. We could, of course, continue with
the sequential builds, but the reason for the second builder is that these
combined builds take very long to generate.
> > > > The Dependent scheduler (which, BTW is still the v3.6 version of
the documentation, not the v3.8 version) seems to fit the bill for what we
need, *except* that it depends on *all* the builds triggered by the initial
scheduler, not just specific builders in that set. That is not desirable in
our scenario.
> > > >
> > > > What we need is something like
> > > >
> > > > --------------
> > > >
> > > > from buildbot.plugins import schedulers
> > > > build = schedulers.SingleBranchScheduler(name="build",
> > > > treeStableTimer=5*60,
> > > > builderNames=["builder1",
"builder2", "builder3", ..., "builderN"])
> > > > second_step = schedulers.Dependent(name="second-step"",
> > > > upstream=build,
> > > > upstream_builders=["builder1",
"builder2"]
> > > > builderNames=["second_step_builder"])
> > > > ---------------
> > > >
> > > > This would trigger "second-step" when both "builder1" and
"builder2" have completed their jobs, ignoring the state of the other
builders.
> > > >
> > > > Is there any way to accomplish this currently, either directly, or
by creating a subclass of Dependent?
> > > >
> > > >
> > > > There is, however, if I read the documentation correctly, a
problem with Dependent: It sounds like there is no transfer of build
properties from the upstream builders, which is a problem, since those
properties are necessary to inform the second step builder where to locate
the data needed for its build.
> > > >
> > > >
> > > > A different method that might work better could be a scheduler
that triggers when it has received all Triggers from specific builders in a
particular scheduler job
> > > >
> > > >
> > > > Any thoughts about this possibility?
> > > >
> > > > _______________________________________________
> > > users mailing list
> > > users at buildbot.net
> > > https://lists.buildbot.net/mailman/listinfo/users
> >
--
Sincerely,
Yngve N. Pettersen
Vivaldi Technologies AS
More information about the users
mailing list