[users at bb.net] Trigger new build/scheduler when two related builds are finished

[Yngve N. Pettersen]

Hello all,

I have now created a variant Triggered scheduler that will wait for 
multiple specific dependent builders for a given buildset.

However, it looks like in base.BaseScheduler instances  
self.master.db.builds.getBuild() return None for all queries, even for an 
existing parent_buildid.

I just completed a debug run that showed that in my special steps.Trigger 
instance, correct information is available from that call, while in the 
call to sch.trigger inside steps.Trigger.run() the data were not available, 
inside the same call sequence.

I suspect the same is the case with 
self.master.db.buildrequests.getBuildRequest, used to get the buildset id, 
which never gets called because the getBuild() does not return valid data.

The buildset ID is necessary to run a wait for the set to complete, in case 
one of the builds it is waiting for fails (or was never started).

Any ideas about how to solve this issue?

On Monday 31. July 2023 14:37:42 (+02:00), Yngve N. Pettersen wrote:

 > 
 > Hi,
 > 
 > Thanks for the suggestion.
 > 
 > However, due to implementation details, specifically build version 
generation, I am not sure it is practical, because there are half a dozen 
other builds being triggered at the same time, by the same scheduler ( and 
the build version generation and retrieval depends on the scheduler name), 
and all those builds must have the same build number (although I can think 
of ways to fake it).
 > 
 > Additionally, I suspect that the information transfer used by the 
universal step is not possible with that method (the data is generated by 
the build jobs, and is unique for each job).
 > 
 > Another practical issue is that this method would block the worker where 
the universal step is done for at least 3 hours, preventing other tasks it 
might need to do from starting until that job is complete.
 > 
 > On Monday 31. July 2023 13:44:03 (+02:00), Roland van Laar wrote:
 > 
 > > Hi,
 > > > To summarize, the desired workflow is:
 > > > - kick off two builds: arm64 and x64
 > > - combine output into one universal installer.
 > > > My advice use a trigger[1] with `waitForFinish`.
 > > > build = BuildFactory()
 > > build.addStep(steps.Trigger(schedulerNames=["buildx64", "buildarm64"], 
waitForFinish=True)
 > > build.addStep()
 > > > Regards,
 > > > Roland
 > > > [1] 
https://docs.buildbot.net/latest/manual/configuration/steps/trigger.html#trigger
 
> > > > On 31-07-2023 11:00, Yngve N. Pettersen wrote:
 > > > Hello all,
 > > >
 > > > We have a build sequence that consists of two separate builds that 
are then combined into a single product (specifically, this concerns Mac 
x64 and arm64 builds being used to generate a universal DMG installer)
 > > >
 > > > At present we are building both sequentially on the same build 
worker which makes the system work, but we are now preparing a new worker 
that will be used for the arm64 build.
 > > >
 > > > The plan is to start both builds at the same time, but that prevents 
the transfer of the (build properties) data necessary to combine the 
products into a universal installer. We could, of course, continue with the 
sequential builds, but the reason for the second builder is that these 
combined builds take very long to generate.
 > > > The Dependent scheduler  (which, BTW is still the v3.6 version of 
the documentation, not the v3.8 version) seems to fit the bill for what we 
need, *except* that it depends on *all* the builds triggered by the initial 
scheduler, not just specific builders in that set. That is not desirable in 
our scenario.
 > > >
 > > > What we need is something like
 > > >
 > > > --------------
 > > >
 > > > from buildbot.plugins import schedulers
 > > > build = schedulers.SingleBranchScheduler(name="build",
 > > >                                          treeStableTimer=5*60,
 > > >                                          builderNames=["builder1", 
"builder2", "builder3", ..., "builderN"])
 > > > second_step = schedulers.Dependent(name="second-step"",
 > > >                                upstream=build,
 > > >                                upstream_builders=["builder1", 
"builder2"]
 > > > builderNames=["second_step_builder"])
 > > > ---------------
 > > >
 > > > This would trigger "second-step" when both "builder1" and "builder2" 
have completed their jobs, ignoring the state of the other builders.
 > > >
 > > > Is there any way to accomplish this currently, either directly, or 
by creating a subclass of Dependent?
 > > >
 > > >
 > > > There is, however, if I read the documentation correctly, a problem 
with Dependent: It sounds like there is no transfer of build properties 
from the upstream builders, which is a problem, since those properties are 
necessary to inform the second step builder where to locate the data needed 
for its build.
 > > >
 > > >
 > > > A different method that might work better could be a scheduler that 
triggers when it has received all Triggers from specific builders in a 
particular scheduler job
 > > >
 > > >
 > > > Any thoughts about this possibility?
 > > >
 > > > _______________________________________________
 > > users mailing list
 > > users at buildbot.net
 > > https://lists.buildbot.net/mailman/listinfo/users
 > 

-- 
Sincerely,
Yngve N. Pettersen
Vivaldi Technologies AS