[users at bb.net] Limit amount of DockerLatentWorkers running on a particular physical machine

[Povilas Kanapickas]

Hi,

If you use renderable *builder* locks then they will not cause builds to
wait for the locks. The builder lock resolution happens before the build
actually starts, even before the canStartBuild function.

Starting a build that later can't acquire the builder locks we just
checked should be rare occurrence. If that's not the case, it's a bug I
would be interested in investigating.

So what I would do is to set builder locks argument to a renderable
function, within that function I would check interesting build
properties such as which worker the build is about to start on and then
return a set of locks that must be acquired. This way you would have
almost complete flexibility. For each resource you can have separate
master lock with maxCount representing maximum resource utilization
(e.g. 512GB RAM or whatever) and then the builders would take e.g.
lock.access('counting', count=8) to acquire a 8GB ram slice).

Regards,
Povilas

On 2022-07-04 21:39, Vlad Bogolin wrote:
> Hi,
> 
> Thank you for your reply! Is there any way to customize what
> oversubscribed means? We already use a locking mechanism, but still this
> translates into having multiple running builds that just wait for the
> locks for several hours. Ideally, I would like to avoid this.
> 
> Also, by any chance, can you read a lock value from the canStartBuild
> function?
> 
> Thank you!
> Vlad Bogolin 
> 
> On Fri, Jul 1, 2022 at 12:40 PM Povilas Kanapickas <povilas at radix.lt
> <mailto:povilas at radix.lt>> wrote:
> 
>     Hi Vlad,
> 
>     You could setup a number of master locks that are each assigned to a
>     particular physical machine. Then you can setup renderable locks for
>     builds: a build would look into what physical machine it's about to
>     launch on and select the correct lock. If the physical machine is
>     oversubscribed, Buildbot will notice that lock can not be acquired look
>     for another worker for the build.
> 
>     Regards,
>     Povilas
> 
>     On 2022-06-28 12:44, Vlad Bogolin wrote:
>     > Hello,
>     >
>     > We are using buildbot with primary DockerLatentWorkers for our CI. So,
>     > given a physical machine, we have several DockerLatentWorkers that may
>     > run on it. While this works well, in some cases buildbot starts
>     too many
>     > latent workers on the same machine. Is there a way to limit starting
>     > builds for a particular DockerLatentWorker if others are already
>     running
>     > on the same machine?
>     >
>     > I feel like this should be achievable using the canStartBuild, but
>     I am
>     > unsure how. Is it possible to access the full list of defined latent
>     > workers and see if one is on or not in the canStartBuild function?
>     >
>     > Thank you!
>     > Vlad Bogolin
>     >
>     > _______________________________________________
>     > users mailing list
>     > users at buildbot.net <mailto:users at buildbot.net>
>     > https://lists.buildbot.net/mailman/listinfo/users
>     <https://lists.buildbot.net/mailman/listinfo/users>
>     >
> 
> 
> 
> -- 
> Vlad