[devel at bb.net] Is it possible to support parallel steps from a single build?

[Yixing Jiang]

My friend Tong (cc’ed) and I will start working on this. 

> On Aug 18, 2019, at 2:36 PM, Yixing Jiang <johnestar.wag at gmail.com> wrote:
> 
> Thanks! This is very helpful. 
> Yeah simply creating a ParallelSteps sounds like a promising direction to go to. 
> 
>> On Aug 18, 2019, at 10:50 AM, Pierre Tardy <tardyp at gmail.com> wrote:
>> 
>> It is indeed not supported, but indeed could be useful.
>> 
>> This is not such a big conceptual change. There is nothing in the
>> protocol that would prevent sending several RemoteCommands at the same
>> time.
>> One could create a ParallelSteps step, which you would take a list of
>> steps to run in parallel.
>> Then, the step would wait that all the other steps are finished before
>> starting the remaining steps.
>> 
>> If you which to try it, we would gladly help you to polish a merge request
>> 
>> Regards
>> Pierre
>> 
>> Le dim. 18 août 2019 à 14:57, Clément Hurlin
>> <clement.hurlin at provenrun.com> a écrit :
>>> 
>>> Indeed, introducing new builders may pollute the list of builders, which I used to think as a bot's public interface. You could maybe prefix them with a label ([internal]) and make sure they show up at the end of the list ?
>>> 
>>> I have never investigated the code of buildbot that makes the interface between builders and steps; but moving from sequential steps to possibly parallel ones seems like a big conceptual change.
>>> 
>>> Clément
>>> 
>>> Thanks Clement, that’s what I read online as well. But I feel like having multiple builders make it not quite flexible, and may confuse our users.
>>> Is it easy to enable parallel steps? If I want to do a pull request for it, do you have any suggestions?
>>> 
>>> On Aug 16, 2019, at 12:48 AM, Clément Hurlin <clement.hurlin at provenrun.com> wrote:
>>> 
>>> Hi Yixing,
>>> 
>>> I believe it is not possible to parallelize steps within a single builder. However, if you're OK to have multiple builders; you could move each parallelizable step to another builder and trigger all these spinoff builders from the main builder.
>>> 
>>> Best,
>>> 
>>> Clément
>>> 
>>> 
>>> Hi there,
>>> 
>>> We are using buildbot for CI tests. In our build we have 20 steps but most of them could be fully parallelized. We wonder if we can allow parallel steps from the single build, this will allow us to run the entire build much faster.
>>> 
>>> Thanks,
>>> Yixing
>>> _______________________________________________
>>> devel mailing list
>>> devel at buildbot.net
>>> https://lists.buildbot.net/mailman/listinfo/devel
>>> 
>>> 
>>> _______________________________________________
>>> devel mailing list
>>> devel at buildbot.net
>>> https://lists.buildbot.net/mailman/listinfo/devel
>