<div dir="ltr">Hi,<div><br></div><div>I have a follow up question: is it possible to use the locking mechanism in a multi-master setup? From my attempts it seems that each master process has its own locks. Do you have any suggestions on what we might use to limit the amount of DockerLatentWorkers that might run if they are shared among multiple master processes?</div><div><br></div><div>Thank you!</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 4, 2022 at 10:20 PM Vlad Bogolin <<a href="mailto:vlad@mariadb.org">vlad@mariadb.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Great, thanks! Please keep me posted!<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 4, 2022 at 10:18 PM Povilas Kanapickas <<a href="mailto:povilas@radix.lt" target="_blank">povilas@radix.lt</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Your setup is exactly what I described and should work (at least in<br>
theory). I will look into this when I have time.<br>
<br>
On 2022-07-04 22:05, Vlad Bogolin wrote:<br>
> Thanks for the prompt reply! <br>
> <br>
> Our locks are defined as you can see<br>
> here <a href="https://github.com/MariaDB/buildbot/blob/main/locks.py" rel="noreferrer" target="_blank">https://github.com/MariaDB/buildbot/blob/main/locks.py</a><br>
> <<a href="https://github.com/MariaDB/buildbot/blob/main/locks.py" rel="noreferrer" target="_blank">https://github.com/MariaDB/buildbot/blob/main/locks.py</a>> and then each<br>
> build receives as argument the lock<br>
> function <a href="https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961" rel="noreferrer" target="_blank">https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961</a><br>
> <<a href="https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961" rel="noreferrer" target="_blank">https://github.com/MariaDB/buildbot/blob/cd5378a7b6549e3bf5930306c2eed29239aa3a38/master.cfg#L961</a>>.<br>
> However, for example this build<br>
> (<a href="https://buildbot.mariadb.org/#/builders/348/builds/1638" rel="noreferrer" target="_blank">https://buildbot.mariadb.org/#/builders/348/builds/1638</a><br>
> <<a href="https://buildbot.mariadb.org/#/builders/348/builds/1638" rel="noreferrer" target="_blank">https://buildbot.mariadb.org/#/builders/348/builds/1638</a>>) is now<br>
> waiting for 6h to acquire the locks. Overall there are 11 builds that<br>
> have started and are waiting for locks currently. I have considered this<br>
> to be normal, but if you think there is an issue please let me know.<br>
> <br>
> Thank you!<br>
> <br>
> On Mon, Jul 4, 2022 at 9:52 PM Povilas Kanapickas <<a href="mailto:povilas@radix.lt" target="_blank">povilas@radix.lt</a><br>
> <mailto:<a href="mailto:povilas@radix.lt" target="_blank">povilas@radix.lt</a>>> wrote:<br>
> <br>
> Hi,<br>
> <br>
> If you use renderable *builder* locks then they will not cause builds to<br>
> wait for the locks. The builder lock resolution happens before the build<br>
> actually starts, even before the canStartBuild function.<br>
> <br>
> Starting a build that later can't acquire the builder locks we just<br>
> checked should be rare occurrence. If that's not the case, it's a bug I<br>
> would be interested in investigating.<br>
> <br>
> So what I would do is to set builder locks argument to a renderable<br>
> function, within that function I would check interesting build<br>
> properties such as which worker the build is about to start on and then<br>
> return a set of locks that must be acquired. This way you would have<br>
> almost complete flexibility. For each resource you can have separate<br>
> master lock with maxCount representing maximum resource utilization<br>
> (e.g. 512GB RAM or whatever) and then the builders would take e.g.<br>
> lock.access('counting', count=8) to acquire a 8GB ram slice).<br>
> <br>
> Regards,<br>
> Povilas<br>
> <br>
> On 2022-07-04 21:39, Vlad Bogolin wrote:<br>
> > Hi,<br>
> ><br>
> > Thank you for your reply! Is there any way to customize what<br>
> > oversubscribed means? We already use a locking mechanism, but<br>
> still this<br>
> > translates into having multiple running builds that just wait for the<br>
> > locks for several hours. Ideally, I would like to avoid this.<br>
> ><br>
> > Also, by any chance, can you read a lock value from the canStartBuild<br>
> > function?<br>
> ><br>
> > Thank you!<br>
> > Vlad Bogolin <br>
> ><br>
> > On Fri, Jul 1, 2022 at 12:40 PM Povilas Kanapickas<br>
> <<a href="mailto:povilas@radix.lt" target="_blank">povilas@radix.lt</a> <mailto:<a href="mailto:povilas@radix.lt" target="_blank">povilas@radix.lt</a>><br>
> > <mailto:<a href="mailto:povilas@radix.lt" target="_blank">povilas@radix.lt</a> <mailto:<a href="mailto:povilas@radix.lt" target="_blank">povilas@radix.lt</a>>>> wrote:<br>
> ><br>
> > Hi Vlad,<br>
> ><br>
> > You could setup a number of master locks that are each<br>
> assigned to a<br>
> > particular physical machine. Then you can setup renderable<br>
> locks for<br>
> > builds: a build would look into what physical machine it's<br>
> about to<br>
> > launch on and select the correct lock. If the physical machine is<br>
> > oversubscribed, Buildbot will notice that lock can not be<br>
> acquired look<br>
> > for another worker for the build.<br>
> ><br>
> > Regards,<br>
> > Povilas<br>
> ><br>
> > On 2022-06-28 12:44, Vlad Bogolin wrote:<br>
> > > Hello,<br>
> > ><br>
> > > We are using buildbot with primary DockerLatentWorkers for<br>
> our CI. So,<br>
> > > given a physical machine, we have several<br>
> DockerLatentWorkers that may<br>
> > > run on it. While this works well, in some cases buildbot starts<br>
> > too many<br>
> > > latent workers on the same machine. Is there a way to limit<br>
> starting<br>
> > > builds for a particular DockerLatentWorker if others are already<br>
> > running<br>
> > > on the same machine?<br>
> > ><br>
> > > I feel like this should be achievable using the<br>
> canStartBuild, but<br>
> > I am<br>
> > > unsure how. Is it possible to access the full list of<br>
> defined latent<br>
> > > workers and see if one is on or not in the canStartBuild<br>
> function?<br>
> > ><br>
> > > Thank you!<br>
> > > Vlad Bogolin<br>
> > ><br>
> > > _______________________________________________<br>
> > > users mailing list<br>
> > > <a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a> <mailto:<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a>><br>
> <mailto:<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a> <mailto:<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a>>><br>
> > > <a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br>
> <<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a>><br>
> > <<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br>
> <<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a>>><br>
> > ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Vlad<br>
> <br>
> <br>
> <br>
> -- <br>
> Vlad<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr">Vlad</div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Vlad</div></div>