[users at bb.net] Limit amount of DockerLatentWorkers running on a particular physical machine

[Vlad Bogolin]

Thanks for the prompt reply!

Our locks are defined as you can see here
https://github.com/MariaDB/buildbot/blob/main/locks.py and then each build
receives as argument the lock function
https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961.
However, for example this build (
https://buildbot.mariadb.org/#/builders/348/builds/1638) is now waiting for
6h to acquire the locks. Overall there are 11 builds that have started and
are waiting for locks currently. I have considered this to be normal, but
if you think there is an issue please let me know.

Thank you!

On Mon, Jul 4, 2022 at 9:52 PM Povilas Kanapickas <povilas at radix.lt> wrote:

> Hi,
>
> If you use renderable *builder* locks then they will not cause builds to
> wait for the locks. The builder lock resolution happens before the build
> actually starts, even before the canStartBuild function.
>
> Starting a build that later can't acquire the builder locks we just
> checked should be rare occurrence. If that's not the case, it's a bug I
> would be interested in investigating.
>
> So what I would do is to set builder locks argument to a renderable
> function, within that function I would check interesting build
> properties such as which worker the build is about to start on and then
> return a set of locks that must be acquired. This way you would have
> almost complete flexibility. For each resource you can have separate
> master lock with maxCount representing maximum resource utilization
> (e.g. 512GB RAM or whatever) and then the builders would take e.g.
> lock.access('counting', count=8) to acquire a 8GB ram slice).
>
> Regards,
> Povilas
>
> On 2022-07-04 21:39, Vlad Bogolin wrote:
> > Hi,
> >
> > Thank you for your reply! Is there any way to customize what
> > oversubscribed means? We already use a locking mechanism, but still this
> > translates into having multiple running builds that just wait for the
> > locks for several hours. Ideally, I would like to avoid this.
> >
> > Also, by any chance, can you read a lock value from the canStartBuild
> > function?
> >
> > Thank you!
> > Vlad Bogolin
> >
> > On Fri, Jul 1, 2022 at 12:40 PM Povilas Kanapickas <povilas at radix.lt
> > <mailto:povilas at radix.lt>> wrote:
> >
> >     Hi Vlad,
> >
> >     You could setup a number of master locks that are each assigned to a
> >     particular physical machine. Then you can setup renderable locks for
> >     builds: a build would look into what physical machine it's about to
> >     launch on and select the correct lock. If the physical machine is
> >     oversubscribed, Buildbot will notice that lock can not be acquired
> look
> >     for another worker for the build.
> >
> >     Regards,
> >     Povilas
> >
> >     On 2022-06-28 12:44, Vlad Bogolin wrote:
> >     > Hello,
> >     >
> >     > We are using buildbot with primary DockerLatentWorkers for our CI.
> So,
> >     > given a physical machine, we have several DockerLatentWorkers that
> may
> >     > run on it. While this works well, in some cases buildbot starts
> >     too many
> >     > latent workers on the same machine. Is there a way to limit
> starting
> >     > builds for a particular DockerLatentWorker if others are already
> >     running
> >     > on the same machine?
> >     >
> >     > I feel like this should be achievable using the canStartBuild, but
> >     I am
> >     > unsure how. Is it possible to access the full list of defined
> latent
> >     > workers and see if one is on or not in the canStartBuild function?
> >     >
> >     > Thank you!
> >     > Vlad Bogolin
> >     >
> >     > _______________________________________________
> >     > users mailing list
> >     > users at buildbot.net <mailto:users at buildbot.net>
> >     > https://lists.buildbot.net/mailman/listinfo/users
> >     <https://lists.buildbot.net/mailman/listinfo/users>
> >     >
> >
> >
> >
> > --
> > Vlad
>


-- 
Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20220704/3bf01d67/attachment.htm>