From povilas at radix.lt  Fri Jul  1 09:40:39 2022
From: povilas at radix.lt (Povilas Kanapickas)
Date: Fri, 1 Jul 2022 12:40:39 +0300
Subject: [users@bb.net] Limit amount of DockerLatentWorkers running on a
 particular physical machine
In-Reply-To: <CADq3Nvf=jqWCxRcjEN_hnSo7aXM_esxxQgvTYxgj+zJBAHV6Nw@mail.gmail.com>
References: <CADq3Nvf=jqWCxRcjEN_hnSo7aXM_esxxQgvTYxgj+zJBAHV6Nw@mail.gmail.com>
Message-ID: <febc1864-fac9-4332-9633-b7b223e59a40@radix.lt>

Hi Vlad,

You could setup a number of master locks that are each assigned to a
particular physical machine. Then you can setup renderable locks for
builds: a build would look into what physical machine it's about to
launch on and select the correct lock. If the physical machine is
oversubscribed, Buildbot will notice that lock can not be acquired look
for another worker for the build.

Regards,
Povilas

On 2022-06-28 12:44, Vlad Bogolin wrote:
> Hello,
> 
> We are using buildbot with primary DockerLatentWorkers for our CI. So,
> given a physical machine, we have several DockerLatentWorkers that may
> run on it. While this works well, in some cases buildbot starts too many
> latent workers on the same machine. Is there a way to limit starting
> builds for a particular DockerLatentWorker if others are already running
> on the same machine?
> 
> I feel like this should be achievable using the canStartBuild, but I am
> unsure how. Is it possible to access the full list of defined latent
> workers and see if one is on or not in the canStartBuild function?
> 
> Thank you!
> Vlad Bogolin
> 
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
> 

From vlad at mariadb.org  Mon Jul  4 18:39:09 2022
From: vlad at mariadb.org (Vlad Bogolin)
Date: Mon, 4 Jul 2022 21:39:09 +0300
Subject: [users@bb.net] Limit amount of DockerLatentWorkers running on a
 particular physical machine
In-Reply-To: <febc1864-fac9-4332-9633-b7b223e59a40@radix.lt>
References: <CADq3Nvf=jqWCxRcjEN_hnSo7aXM_esxxQgvTYxgj+zJBAHV6Nw@mail.gmail.com>
 <febc1864-fac9-4332-9633-b7b223e59a40@radix.lt>
Message-ID: <CADq3NvekV=VU_kaXob6eCcejBeHMmoO0YX0_G5RRtn-LDvzJ5w@mail.gmail.com>

Hi,

Thank you for your reply! Is there any way to customize what oversubscribed
means? We already use a locking mechanism, but still this translates into
having multiple running builds that just wait for the locks for several
hours. Ideally, I would like to avoid this.

Also, by any chance, can you read a lock value from the canStartBuild
function?

Thank you!
Vlad Bogolin

On Fri, Jul 1, 2022 at 12:40 PM Povilas Kanapickas <povilas at radix.lt> wrote:

> Hi Vlad,
>
> You could setup a number of master locks that are each assigned to a
> particular physical machine. Then you can setup renderable locks for
> builds: a build would look into what physical machine it's about to
> launch on and select the correct lock. If the physical machine is
> oversubscribed, Buildbot will notice that lock can not be acquired look
> for another worker for the build.
>
> Regards,
> Povilas
>
> On 2022-06-28 12:44, Vlad Bogolin wrote:
> > Hello,
> >
> > We are using buildbot with primary DockerLatentWorkers for our CI. So,
> > given a physical machine, we have several DockerLatentWorkers that may
> > run on it. While this works well, in some cases buildbot starts too many
> > latent workers on the same machine. Is there a way to limit starting
> > builds for a particular DockerLatentWorker if others are already running
> > on the same machine?
> >
> > I feel like this should be achievable using the canStartBuild, but I am
> > unsure how. Is it possible to access the full list of defined latent
> > workers and see if one is on or not in the canStartBuild function?
> >
> > Thank you!
> > Vlad Bogolin
> >
> > _______________________________________________
> > users mailing list
> > users at buildbot.net
> > https://lists.buildbot.net/mailman/listinfo/users
> >
>


-- 
Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20220704/5ed737ef/attachment.htm>

From povilas at radix.lt  Mon Jul  4 18:52:39 2022
From: povilas at radix.lt (Povilas Kanapickas)
Date: Mon, 4 Jul 2022 21:52:39 +0300
Subject: [users@bb.net] Limit amount of DockerLatentWorkers running on a
 particular physical machine
In-Reply-To: <CADq3NvekV=VU_kaXob6eCcejBeHMmoO0YX0_G5RRtn-LDvzJ5w@mail.gmail.com>
References: <CADq3Nvf=jqWCxRcjEN_hnSo7aXM_esxxQgvTYxgj+zJBAHV6Nw@mail.gmail.com>
 <febc1864-fac9-4332-9633-b7b223e59a40@radix.lt>
 <CADq3NvekV=VU_kaXob6eCcejBeHMmoO0YX0_G5RRtn-LDvzJ5w@mail.gmail.com>
Message-ID: <ae21b04b-c316-62f8-907d-fff9298f5da7@radix.lt>

Hi,

If you use renderable *builder* locks then they will not cause builds to
wait for the locks. The builder lock resolution happens before the build
actually starts, even before the canStartBuild function.

Starting a build that later can't acquire the builder locks we just
checked should be rare occurrence. If that's not the case, it's a bug I
would be interested in investigating.

So what I would do is to set builder locks argument to a renderable
function, within that function I would check interesting build
properties such as which worker the build is about to start on and then
return a set of locks that must be acquired. This way you would have
almost complete flexibility. For each resource you can have separate
master lock with maxCount representing maximum resource utilization
(e.g. 512GB RAM or whatever) and then the builders would take e.g.
lock.access('counting', count=8) to acquire a 8GB ram slice).

Regards,
Povilas

On 2022-07-04 21:39, Vlad Bogolin wrote:
> Hi,
> 
> Thank you?for your reply! Is there any way to customize what
> oversubscribed means? We already use a locking mechanism, but still this
> translates into having multiple running builds?that just wait for the
> locks for several hours. Ideally, I would like to avoid this.
> 
> Also, by any chance, can you read a lock value from the canStartBuild
> function?
> 
> Thank you!
> Vlad Bogolin?
> 
> On Fri, Jul 1, 2022 at 12:40 PM Povilas Kanapickas <povilas at radix.lt
> <mailto:povilas at radix.lt>> wrote:
> 
>     Hi Vlad,
> 
>     You could setup a number of master locks that are each assigned to a
>     particular physical machine. Then you can setup renderable locks for
>     builds: a build would look into what physical machine it's about to
>     launch on and select the correct lock. If the physical machine is
>     oversubscribed, Buildbot will notice that lock can not be acquired look
>     for another worker for the build.
> 
>     Regards,
>     Povilas
> 
>     On 2022-06-28 12:44, Vlad Bogolin wrote:
>     > Hello,
>     >
>     > We are using buildbot with primary DockerLatentWorkers for our CI. So,
>     > given a physical machine, we have several DockerLatentWorkers that may
>     > run on it. While this works well, in some cases buildbot starts
>     too many
>     > latent workers on the same machine. Is there a way to limit starting
>     > builds for a particular DockerLatentWorker if others are already
>     running
>     > on the same machine?
>     >
>     > I feel like this should be achievable using the canStartBuild, but
>     I am
>     > unsure how. Is it possible to access the full list of defined latent
>     > workers and see if one is on or not in the canStartBuild function?
>     >
>     > Thank you!
>     > Vlad Bogolin
>     >
>     > _______________________________________________
>     > users mailing list
>     > users at buildbot.net <mailto:users at buildbot.net>
>     > https://lists.buildbot.net/mailman/listinfo/users
>     <https://lists.buildbot.net/mailman/listinfo/users>
>     >
> 
> 
> 
> -- 
> Vlad

From vlad at mariadb.org  Mon Jul  4 19:05:48 2022
From: vlad at mariadb.org (Vlad Bogolin)
Date: Mon, 4 Jul 2022 22:05:48 +0300
Subject: [users@bb.net] Limit amount of DockerLatentWorkers running on a
 particular physical machine
In-Reply-To: <ae21b04b-c316-62f8-907d-fff9298f5da7@radix.lt>
References: <CADq3Nvf=jqWCxRcjEN_hnSo7aXM_esxxQgvTYxgj+zJBAHV6Nw@mail.gmail.com>
 <febc1864-fac9-4332-9633-b7b223e59a40@radix.lt>
 <CADq3NvekV=VU_kaXob6eCcejBeHMmoO0YX0_G5RRtn-LDvzJ5w@mail.gmail.com>
 <ae21b04b-c316-62f8-907d-fff9298f5da7@radix.lt>
Message-ID: <CADq3Nvdo6BmmmvktRjW-BcYgoxiZu_2SBLXtq20Ng6tSqw2+SA@mail.gmail.com>

Thanks for the prompt reply!

Our locks are defined as you can see here
https://github.com/MariaDB/buildbot/blob/main/locks.py and then each build
receives as argument the lock function
https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961.
However, for example this build (
https://buildbot.mariadb.org/#/builders/348/builds/1638) is now waiting for
6h to acquire the locks. Overall there are 11 builds that have started and
are waiting for locks currently. I have considered this to be normal, but
if you think there is an issue please let me know.

Thank you!

On Mon, Jul 4, 2022 at 9:52 PM Povilas Kanapickas <povilas at radix.lt> wrote:

> Hi,
>
> If you use renderable *builder* locks then they will not cause builds to
> wait for the locks. The builder lock resolution happens before the build
> actually starts, even before the canStartBuild function.
>
> Starting a build that later can't acquire the builder locks we just
> checked should be rare occurrence. If that's not the case, it's a bug I
> would be interested in investigating.
>
> So what I would do is to set builder locks argument to a renderable
> function, within that function I would check interesting build
> properties such as which worker the build is about to start on and then
> return a set of locks that must be acquired. This way you would have
> almost complete flexibility. For each resource you can have separate
> master lock with maxCount representing maximum resource utilization
> (e.g. 512GB RAM or whatever) and then the builders would take e.g.
> lock.access('counting', count=8) to acquire a 8GB ram slice).
>
> Regards,
> Povilas
>
> On 2022-07-04 21:39, Vlad Bogolin wrote:
> > Hi,
> >
> > Thank you for your reply! Is there any way to customize what
> > oversubscribed means? We already use a locking mechanism, but still this
> > translates into having multiple running builds that just wait for the
> > locks for several hours. Ideally, I would like to avoid this.
> >
> > Also, by any chance, can you read a lock value from the canStartBuild
> > function?
> >
> > Thank you!
> > Vlad Bogolin
> >
> > On Fri, Jul 1, 2022 at 12:40 PM Povilas Kanapickas <povilas at radix.lt
> > <mailto:povilas at radix.lt>> wrote:
> >
> >     Hi Vlad,
> >
> >     You could setup a number of master locks that are each assigned to a
> >     particular physical machine. Then you can setup renderable locks for
> >     builds: a build would look into what physical machine it's about to
> >     launch on and select the correct lock. If the physical machine is
> >     oversubscribed, Buildbot will notice that lock can not be acquired
> look
> >     for another worker for the build.
> >
> >     Regards,
> >     Povilas
> >
> >     On 2022-06-28 12:44, Vlad Bogolin wrote:
> >     > Hello,
> >     >
> >     > We are using buildbot with primary DockerLatentWorkers for our CI.
> So,
> >     > given a physical machine, we have several DockerLatentWorkers that
> may
> >     > run on it. While this works well, in some cases buildbot starts
> >     too many
> >     > latent workers on the same machine. Is there a way to limit
> starting
> >     > builds for a particular DockerLatentWorker if others are already
> >     running
> >     > on the same machine?
> >     >
> >     > I feel like this should be achievable using the canStartBuild, but
> >     I am
> >     > unsure how. Is it possible to access the full list of defined
> latent
> >     > workers and see if one is on or not in the canStartBuild function?
> >     >
> >     > Thank you!
> >     > Vlad Bogolin
> >     >
> >     > _______________________________________________
> >     > users mailing list
> >     > users at buildbot.net <mailto:users at buildbot.net>
> >     > https://lists.buildbot.net/mailman/listinfo/users
> >     <https://lists.buildbot.net/mailman/listinfo/users>
> >     >
> >
> >
> >
> > --
> > Vlad
>


-- 
Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20220704/3bf01d67/attachment.htm>

From povilas at radix.lt  Mon Jul  4 19:18:13 2022
From: povilas at radix.lt (Povilas Kanapickas)
Date: Mon, 4 Jul 2022 22:18:13 +0300
Subject: [users@bb.net] Limit amount of DockerLatentWorkers running on a
 particular physical machine
In-Reply-To: <CADq3Nvdo6BmmmvktRjW-BcYgoxiZu_2SBLXtq20Ng6tSqw2+SA@mail.gmail.com>
References: <CADq3Nvf=jqWCxRcjEN_hnSo7aXM_esxxQgvTYxgj+zJBAHV6Nw@mail.gmail.com>
 <febc1864-fac9-4332-9633-b7b223e59a40@radix.lt>
 <CADq3NvekV=VU_kaXob6eCcejBeHMmoO0YX0_G5RRtn-LDvzJ5w@mail.gmail.com>
 <ae21b04b-c316-62f8-907d-fff9298f5da7@radix.lt>
 <CADq3Nvdo6BmmmvktRjW-BcYgoxiZu_2SBLXtq20Ng6tSqw2+SA@mail.gmail.com>
Message-ID: <e24c6dd8-14e2-6f6b-dd9e-a82d64522eef@radix.lt>

Your setup is exactly what I described and should work (at least in
theory). I will look into this when I have time.

On 2022-07-04 22:05, Vlad Bogolin wrote:
> Thanks for the prompt reply!?
> 
> Our locks are defined as you can see
> here?https://github.com/MariaDB/buildbot/blob/main/locks.py
> <https://github.com/MariaDB/buildbot/blob/main/locks.py> and then each
> build receives as argument the lock
> function?https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961
> <https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961>.
> However, for example this build
> (https://buildbot.mariadb.org/#/builders/348/builds/1638
> <https://buildbot.mariadb.org/#/builders/348/builds/1638>) is now
> waiting for 6h to acquire the locks. Overall there are 11 builds that
> have started and are waiting for locks currently. I have considered this
> to be normal, but if you think there is an issue please let me know.
> 
> Thank you!
> 
> On Mon, Jul 4, 2022 at 9:52 PM Povilas Kanapickas <povilas at radix.lt
> <mailto:povilas at radix.lt>> wrote:
> 
>     Hi,
> 
>     If you use renderable *builder* locks then they will not cause builds to
>     wait for the locks. The builder lock resolution happens before the build
>     actually starts, even before the canStartBuild function.
> 
>     Starting a build that later can't acquire the builder locks we just
>     checked should be rare occurrence. If that's not the case, it's a bug I
>     would be interested in investigating.
> 
>     So what I would do is to set builder locks argument to a renderable
>     function, within that function I would check interesting build
>     properties such as which worker the build is about to start on and then
>     return a set of locks that must be acquired. This way you would have
>     almost complete flexibility. For each resource you can have separate
>     master lock with maxCount representing maximum resource utilization
>     (e.g. 512GB RAM or whatever) and then the builders would take e.g.
>     lock.access('counting', count=8) to acquire a 8GB ram slice).
> 
>     Regards,
>     Povilas
> 
>     On 2022-07-04 21:39, Vlad Bogolin wrote:
>     > Hi,
>     >
>     > Thank you?for your reply! Is there any way to customize what
>     > oversubscribed means? We already use a locking mechanism, but
>     still this
>     > translates into having multiple running builds?that just wait for the
>     > locks for several hours. Ideally, I would like to avoid this.
>     >
>     > Also, by any chance, can you read a lock value from the canStartBuild
>     > function?
>     >
>     > Thank you!
>     > Vlad Bogolin?
>     >
>     > On Fri, Jul 1, 2022 at 12:40 PM Povilas Kanapickas
>     <povilas at radix.lt <mailto:povilas at radix.lt>
>     > <mailto:povilas at radix.lt <mailto:povilas at radix.lt>>> wrote:
>     >
>     >? ? ?Hi Vlad,
>     >
>     >? ? ?You could setup a number of master locks that are each
>     assigned to a
>     >? ? ?particular physical machine. Then you can setup renderable
>     locks for
>     >? ? ?builds: a build would look into what physical machine it's
>     about to
>     >? ? ?launch on and select the correct lock. If the physical machine is
>     >? ? ?oversubscribed, Buildbot will notice that lock can not be
>     acquired look
>     >? ? ?for another worker for the build.
>     >
>     >? ? ?Regards,
>     >? ? ?Povilas
>     >
>     >? ? ?On 2022-06-28 12:44, Vlad Bogolin wrote:
>     >? ? ?> Hello,
>     >? ? ?>
>     >? ? ?> We are using buildbot with primary DockerLatentWorkers for
>     our CI. So,
>     >? ? ?> given a physical machine, we have several
>     DockerLatentWorkers that may
>     >? ? ?> run on it. While this works well, in some cases buildbot starts
>     >? ? ?too many
>     >? ? ?> latent workers on the same machine. Is there a way to limit
>     starting
>     >? ? ?> builds for a particular DockerLatentWorker if others are already
>     >? ? ?running
>     >? ? ?> on the same machine?
>     >? ? ?>
>     >? ? ?> I feel like this should be achievable using the
>     canStartBuild, but
>     >? ? ?I am
>     >? ? ?> unsure how. Is it possible to access the full list of
>     defined latent
>     >? ? ?> workers and see if one is on or not in the canStartBuild
>     function?
>     >? ? ?>
>     >? ? ?> Thank you!
>     >? ? ?> Vlad Bogolin
>     >? ? ?>
>     >? ? ?> _______________________________________________
>     >? ? ?> users mailing list
>     >? ? ?> users at buildbot.net <mailto:users at buildbot.net>
>     <mailto:users at buildbot.net <mailto:users at buildbot.net>>
>     >? ? ?> https://lists.buildbot.net/mailman/listinfo/users
>     <https://lists.buildbot.net/mailman/listinfo/users>
>     >? ? ?<https://lists.buildbot.net/mailman/listinfo/users
>     <https://lists.buildbot.net/mailman/listinfo/users>>
>     >? ? ?>
>     >
>     >
>     >
>     > --
>     > Vlad
> 
> 
> 
> -- 
> Vlad

From vlad at mariadb.org  Mon Jul  4 19:20:21 2022
From: vlad at mariadb.org (Vlad Bogolin)
Date: Mon, 4 Jul 2022 22:20:21 +0300
Subject: [users@bb.net] Limit amount of DockerLatentWorkers running on a
 particular physical machine
In-Reply-To: <e24c6dd8-14e2-6f6b-dd9e-a82d64522eef@radix.lt>
References: <CADq3Nvf=jqWCxRcjEN_hnSo7aXM_esxxQgvTYxgj+zJBAHV6Nw@mail.gmail.com>
 <febc1864-fac9-4332-9633-b7b223e59a40@radix.lt>
 <CADq3NvekV=VU_kaXob6eCcejBeHMmoO0YX0_G5RRtn-LDvzJ5w@mail.gmail.com>
 <ae21b04b-c316-62f8-907d-fff9298f5da7@radix.lt>
 <CADq3Nvdo6BmmmvktRjW-BcYgoxiZu_2SBLXtq20Ng6tSqw2+SA@mail.gmail.com>
 <e24c6dd8-14e2-6f6b-dd9e-a82d64522eef@radix.lt>
Message-ID: <CADq3NvcWpMr7VyOXhROfOwkTSiOsTofV5gVAN3T4GaHkFxY_9A@mail.gmail.com>

Great, thanks! Please keep me posted!

On Mon, Jul 4, 2022 at 10:18 PM Povilas Kanapickas <povilas at radix.lt> wrote:

> Your setup is exactly what I described and should work (at least in
> theory). I will look into this when I have time.
>
> On 2022-07-04 22:05, Vlad Bogolin wrote:
> > Thanks for the prompt reply!
> >
> > Our locks are defined as you can see
> > here https://github.com/MariaDB/buildbot/blob/main/locks.py
> > <https://github.com/MariaDB/buildbot/blob/main/locks.py> and then each
> > build receives as argument the lock
> > function
> https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961
> > <
> https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961
> >.
> > However, for example this build
> > (https://buildbot.mariadb.org/#/builders/348/builds/1638
> > <https://buildbot.mariadb.org/#/builders/348/builds/1638>) is now
> > waiting for 6h to acquire the locks. Overall there are 11 builds that
> > have started and are waiting for locks currently. I have considered this
> > to be normal, but if you think there is an issue please let me know.
> >
> > Thank you!
> >
> > On Mon, Jul 4, 2022 at 9:52 PM Povilas Kanapickas <povilas at radix.lt
> > <mailto:povilas at radix.lt>> wrote:
> >
> >     Hi,
> >
> >     If you use renderable *builder* locks then they will not cause
> builds to
> >     wait for the locks. The builder lock resolution happens before the
> build
> >     actually starts, even before the canStartBuild function.
> >
> >     Starting a build that later can't acquire the builder locks we just
> >     checked should be rare occurrence. If that's not the case, it's a
> bug I
> >     would be interested in investigating.
> >
> >     So what I would do is to set builder locks argument to a renderable
> >     function, within that function I would check interesting build
> >     properties such as which worker the build is about to start on and
> then
> >     return a set of locks that must be acquired. This way you would have
> >     almost complete flexibility. For each resource you can have separate
> >     master lock with maxCount representing maximum resource utilization
> >     (e.g. 512GB RAM or whatever) and then the builders would take e.g.
> >     lock.access('counting', count=8) to acquire a 8GB ram slice).
> >
> >     Regards,
> >     Povilas
> >
> >     On 2022-07-04 21:39, Vlad Bogolin wrote:
> >     > Hi,
> >     >
> >     > Thank you for your reply! Is there any way to customize what
> >     > oversubscribed means? We already use a locking mechanism, but
> >     still this
> >     > translates into having multiple running builds that just wait for
> the
> >     > locks for several hours. Ideally, I would like to avoid this.
> >     >
> >     > Also, by any chance, can you read a lock value from the
> canStartBuild
> >     > function?
> >     >
> >     > Thank you!
> >     > Vlad Bogolin
> >     >
> >     > On Fri, Jul 1, 2022 at 12:40 PM Povilas Kanapickas
> >     <povilas at radix.lt <mailto:povilas at radix.lt>
> >     > <mailto:povilas at radix.lt <mailto:povilas at radix.lt>>> wrote:
> >     >
> >     >     Hi Vlad,
> >     >
> >     >     You could setup a number of master locks that are each
> >     assigned to a
> >     >     particular physical machine. Then you can setup renderable
> >     locks for
> >     >     builds: a build would look into what physical machine it's
> >     about to
> >     >     launch on and select the correct lock. If the physical machine
> is
> >     >     oversubscribed, Buildbot will notice that lock can not be
> >     acquired look
> >     >     for another worker for the build.
> >     >
> >     >     Regards,
> >     >     Povilas
> >     >
> >     >     On 2022-06-28 12:44, Vlad Bogolin wrote:
> >     >     > Hello,
> >     >     >
> >     >     > We are using buildbot with primary DockerLatentWorkers for
> >     our CI. So,
> >     >     > given a physical machine, we have several
> >     DockerLatentWorkers that may
> >     >     > run on it. While this works well, in some cases buildbot
> starts
> >     >     too many
> >     >     > latent workers on the same machine. Is there a way to limit
> >     starting
> >     >     > builds for a particular DockerLatentWorker if others are
> already
> >     >     running
> >     >     > on the same machine?
> >     >     >
> >     >     > I feel like this should be achievable using the
> >     canStartBuild, but
> >     >     I am
> >     >     > unsure how. Is it possible to access the full list of
> >     defined latent
> >     >     > workers and see if one is on or not in the canStartBuild
> >     function?
> >     >     >
> >     >     > Thank you!
> >     >     > Vlad Bogolin
> >     >     >
> >     >     > _______________________________________________
> >     >     > users mailing list
> >     >     > users at buildbot.net <mailto:users at buildbot.net>
> >     <mailto:users at buildbot.net <mailto:users at buildbot.net>>
> >     >     > https://lists.buildbot.net/mailman/listinfo/users
> >     <https://lists.buildbot.net/mailman/listinfo/users>
> >     >     <https://lists.buildbot.net/mailman/listinfo/users
> >     <https://lists.buildbot.net/mailman/listinfo/users>>
> >     >     >
> >     >
> >     >
> >     >
> >     > --
> >     > Vlad
> >
> >
> >
> > --
> > Vlad
>


-- 
Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20220704/f61e72c9/attachment.htm>

From vlad at mariadb.org  Wed Jul  6 10:51:06 2022
From: vlad at mariadb.org (Vlad Bogolin)
Date: Wed, 6 Jul 2022 13:51:06 +0300
Subject: [users@bb.net] Limit amount of DockerLatentWorkers running on a
 particular physical machine
In-Reply-To: <CADq3NvcWpMr7VyOXhROfOwkTSiOsTofV5gVAN3T4GaHkFxY_9A@mail.gmail.com>
References: <CADq3Nvf=jqWCxRcjEN_hnSo7aXM_esxxQgvTYxgj+zJBAHV6Nw@mail.gmail.com>
 <febc1864-fac9-4332-9633-b7b223e59a40@radix.lt>
 <CADq3NvekV=VU_kaXob6eCcejBeHMmoO0YX0_G5RRtn-LDvzJ5w@mail.gmail.com>
 <ae21b04b-c316-62f8-907d-fff9298f5da7@radix.lt>
 <CADq3Nvdo6BmmmvktRjW-BcYgoxiZu_2SBLXtq20Ng6tSqw2+SA@mail.gmail.com>
 <e24c6dd8-14e2-6f6b-dd9e-a82d64522eef@radix.lt>
 <CADq3NvcWpMr7VyOXhROfOwkTSiOsTofV5gVAN3T4GaHkFxY_9A@mail.gmail.com>
Message-ID: <CADq3NvfxLRowq5371A-ANiV8RY4Kp9DrkJrMvTYmvigbRyCRgw@mail.gmail.com>

Hi,

I have a follow up question: is it possible to use the locking mechanism in
a multi-master setup? From my attempts it seems that each master process
has its own locks. Do you have any suggestions on what we might use to
limit the amount of DockerLatentWorkers that might run if they are shared
among multiple master processes?

Thank you!

On Mon, Jul 4, 2022 at 10:20 PM Vlad Bogolin <vlad at mariadb.org> wrote:

> Great, thanks! Please keep me posted!
>
> On Mon, Jul 4, 2022 at 10:18 PM Povilas Kanapickas <povilas at radix.lt>
> wrote:
>
>> Your setup is exactly what I described and should work (at least in
>> theory). I will look into this when I have time.
>>
>> On 2022-07-04 22:05, Vlad Bogolin wrote:
>> > Thanks for the prompt reply!
>> >
>> > Our locks are defined as you can see
>> > here https://github.com/MariaDB/buildbot/blob/main/locks.py
>> > <https://github.com/MariaDB/buildbot/blob/main/locks.py> and then each
>> > build receives as argument the lock
>> > function
>> https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961
>> > <
>> https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961
>> >.
>> > However, for example this build
>> > (https://buildbot.mariadb.org/#/builders/348/builds/1638
>> > <https://buildbot.mariadb.org/#/builders/348/builds/1638>) is now
>> > waiting for 6h to acquire the locks. Overall there are 11 builds that
>> > have started and are waiting for locks currently. I have considered this
>> > to be normal, but if you think there is an issue please let me know.
>> >
>> > Thank you!
>> >
>> > On Mon, Jul 4, 2022 at 9:52 PM Povilas Kanapickas <povilas at radix.lt
>> > <mailto:povilas at radix.lt>> wrote:
>> >
>> >     Hi,
>> >
>> >     If you use renderable *builder* locks then they will not cause
>> builds to
>> >     wait for the locks. The builder lock resolution happens before the
>> build
>> >     actually starts, even before the canStartBuild function.
>> >
>> >     Starting a build that later can't acquire the builder locks we just
>> >     checked should be rare occurrence. If that's not the case, it's a
>> bug I
>> >     would be interested in investigating.
>> >
>> >     So what I would do is to set builder locks argument to a renderable
>> >     function, within that function I would check interesting build
>> >     properties such as which worker the build is about to start on and
>> then
>> >     return a set of locks that must be acquired. This way you would have
>> >     almost complete flexibility. For each resource you can have separate
>> >     master lock with maxCount representing maximum resource utilization
>> >     (e.g. 512GB RAM or whatever) and then the builders would take e.g.
>> >     lock.access('counting', count=8) to acquire a 8GB ram slice).
>> >
>> >     Regards,
>> >     Povilas
>> >
>> >     On 2022-07-04 21:39, Vlad Bogolin wrote:
>> >     > Hi,
>> >     >
>> >     > Thank you for your reply! Is there any way to customize what
>> >     > oversubscribed means? We already use a locking mechanism, but
>> >     still this
>> >     > translates into having multiple running builds that just wait for
>> the
>> >     > locks for several hours. Ideally, I would like to avoid this.
>> >     >
>> >     > Also, by any chance, can you read a lock value from the
>> canStartBuild
>> >     > function?
>> >     >
>> >     > Thank you!
>> >     > Vlad Bogolin
>> >     >
>> >     > On Fri, Jul 1, 2022 at 12:40 PM Povilas Kanapickas
>> >     <povilas at radix.lt <mailto:povilas at radix.lt>
>> >     > <mailto:povilas at radix.lt <mailto:povilas at radix.lt>>> wrote:
>> >     >
>> >     >     Hi Vlad,
>> >     >
>> >     >     You could setup a number of master locks that are each
>> >     assigned to a
>> >     >     particular physical machine. Then you can setup renderable
>> >     locks for
>> >     >     builds: a build would look into what physical machine it's
>> >     about to
>> >     >     launch on and select the correct lock. If the physical
>> machine is
>> >     >     oversubscribed, Buildbot will notice that lock can not be
>> >     acquired look
>> >     >     for another worker for the build.
>> >     >
>> >     >     Regards,
>> >     >     Povilas
>> >     >
>> >     >     On 2022-06-28 12:44, Vlad Bogolin wrote:
>> >     >     > Hello,
>> >     >     >
>> >     >     > We are using buildbot with primary DockerLatentWorkers for
>> >     our CI. So,
>> >     >     > given a physical machine, we have several
>> >     DockerLatentWorkers that may
>> >     >     > run on it. While this works well, in some cases buildbot
>> starts
>> >     >     too many
>> >     >     > latent workers on the same machine. Is there a way to limit
>> >     starting
>> >     >     > builds for a particular DockerLatentWorker if others are
>> already
>> >     >     running
>> >     >     > on the same machine?
>> >     >     >
>> >     >     > I feel like this should be achievable using the
>> >     canStartBuild, but
>> >     >     I am
>> >     >     > unsure how. Is it possible to access the full list of
>> >     defined latent
>> >     >     > workers and see if one is on or not in the canStartBuild
>> >     function?
>> >     >     >
>> >     >     > Thank you!
>> >     >     > Vlad Bogolin
>> >     >     >
>> >     >     > _______________________________________________
>> >     >     > users mailing list
>> >     >     > users at buildbot.net <mailto:users at buildbot.net>
>> >     <mailto:users at buildbot.net <mailto:users at buildbot.net>>
>> >     >     > https://lists.buildbot.net/mailman/listinfo/users
>> >     <https://lists.buildbot.net/mailman/listinfo/users>
>> >     >     <https://lists.buildbot.net/mailman/listinfo/users
>> >     <https://lists.buildbot.net/mailman/listinfo/users>>
>> >     >     >
>> >     >
>> >     >
>> >     >
>> >     > --
>> >     > Vlad
>> >
>> >
>> >
>> > --
>> > Vlad
>>
>
>
> --
> Vlad
>


-- 
Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20220706/f715cc34/attachment.htm>

From h.sorby at auckland.ac.nz  Tue Jul 12 09:13:32 2022
From: h.sorby at auckland.ac.nz (Hugh Sorby)
Date: Tue, 12 Jul 2022 09:13:32 +0000
Subject: [users@bb.net] Get previous step build result
Message-ID: <MEYPR01MB6840F15FDD6DFA5C1AFDB170C4869@MEYPR01MB6840.ausprd01.prod.outlook.com>

I'm using Buildbot 3.5.0 and I would like to know if the previous step failed or succeeded.
I have tried to use the doStepIf method to figure out the previous steps status, but I haven't been able to make this work.

Is it possible to do this with 3.5.0?

Many thanks for any suggestions/pointers.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20220712/e35aeeee/attachment.htm>

From roland at micite.net  Wed Jul 13 12:36:54 2022
From: roland at micite.net (Roland van Laar)
Date: Wed, 13 Jul 2022 14:36:54 +0200
Subject: [users@bb.net] Get previous step build result
In-Reply-To: <MEYPR01MB6840F15FDD6DFA5C1AFDB170C4869@MEYPR01MB6840.ausprd01.prod.outlook.com>
References: <MEYPR01MB6840F15FDD6DFA5C1AFDB170C4869@MEYPR01MB6840.ausprd01.prod.outlook.com>
Message-ID: <3c97ea74-1dd0-dd85-c71b-50074612972e@micite.net>

On 12-07-2022 11:13, Hugh Sorby wrote:
> I'm using Buildbot 3.5.0 and I would like to know if the previous step 
> failed or succeeded.
> I have tried to use the doStepIf method to figure out the previous 
> steps status, but I haven't been able to make this work.
>
> Is it possible to do this with 3.5.0?

Hey,

sorry can't give you a dry cut answer. I do have a few pointers for when 
we needed to fetch a previous build.

Have a look at: `getPreviousBuild` from:
https://github.com/buildbot/buildbot/blob/master/master/buildbot/reporters/utils.py

Over at scummvm we use a custom reporter which compares the current 
build to the last one on master branch.

Here is our vendorized `getPreviousBuild`:
https://github.com/scummvm/scummvm-sites/blob/director-buildbot/director/vendor/reporter_utils.py

Note: we're on 2.8.4

Regards,

Roland


> Many thanks for any suggestions/pointers.
>
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20220713/f8403464/attachment.htm>

From h.sorby at auckland.ac.nz  Thu Jul 14 10:10:32 2022
From: h.sorby at auckland.ac.nz (Hugh Sorby)
Date: Thu, 14 Jul 2022 10:10:32 +0000
Subject: [users@bb.net] Get previous step build result
In-Reply-To: <3c97ea74-1dd0-dd85-c71b-50074612972e@micite.net>
References: <MEYPR01MB6840F15FDD6DFA5C1AFDB170C4869@MEYPR01MB6840.ausprd01.prod.outlook.com>
 <3c97ea74-1dd0-dd85-c71b-50074612972e@micite.net>
Message-ID: <MEYPR01MB684005979E9CF1992034415EC4889@MEYPR01MB6840.ausprd01.prod.outlook.com>

Thanks Roland for the pointer, I ended up with this



@defer.inlineCallbacks

def end_to_end_testing_succeeded(build_step):

    build_id = build_step.build.buildid

    master = build_step.build.master

    build_steps = yield master.data.get(("builds", build_id, 'steps'))

    for step in build_steps:

        if step['name'] == 'Trigger End-to-End Testing':

            return step['results'] == SUCCESS


    return False


And applied it like this:

doStepIf=end_to_end_testing_succeeded,

Seems to work.

________________________________
From: users <users-bounces at buildbot.net> on behalf of Roland van Laar <roland at micite.net>
Sent: 14 July 2022 00:36
To: users at buildbot.net <users at buildbot.net>
Subject: Re: [users at bb.net] Get previous step build result

Caution - Forged External Domain!
This e-mail cannot be validated and may not have been sent by the sender shown in the 'From' field.
If you were not expecting to receive this e-mail we recommend you contact the sender to confirm that they sent it.
If you believe this email was legitimately sent, we suggest the sender notify their e-mail administrator that it has been received as a forged (fake) e-mail by the University of Auckland.
Please contact the Staff Service Centre on extension 86000 if you require further assistance.

On 12-07-2022 11:13, Hugh Sorby wrote:
I'm using Buildbot 3.5.0 and I would like to know if the previous step failed or succeeded.
I have tried to use the doStepIf method to figure out the previous steps status, but I haven't been able to make this work.

Is it possible to do this with 3.5.0?

Hey,

sorry can't give you a dry cut answer. I do have a few pointers for when we needed to fetch a previous build.

Have a look at: `getPreviousBuild` from:
https://github.com/buildbot/buildbot/blob/master/master/buildbot/reporters/utils.py<https://github.com/buildbot/buildbot/blob/master/master/buildbot/reporters/utils.py>

Over at scummvm we use a custom reporter which compares the current build to the last one on master branch.

Here is our vendorized `getPreviousBuild`:
https://github.com/scummvm/scummvm-sites/blob/director-buildbot/director/vendor/reporter_utils.py<https://github.com/scummvm/scummvm-sites/blob/director-buildbot/director/vendor/reporter_utils.py>

Note: we're on 2.8.4

Regards,

Roland


Many thanks for any suggestions/pointers.



_______________________________________________
users mailing list
users at buildbot.net<mailto:users at buildbot.net>
https://lists.buildbot.net/mailman/listinfo/users<https://lists.buildbot.net/mailman/listinfo/users>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20220714/f91aab05/attachment.htm>