[users at bb.net] Performance issues in 2.10

Pierre Tardy tardyp at gmail.com
Tue Jan 12 09:59:42 UTC 2021


Hello,

A lot of things happen between 2.7 and 2.10, although I don't see anything
which could impact the performance that much. (maybe new reporter
framework, but really not convinced)
If you see that the db is underutilized this must be a classical reactor
starvation.
With asynchronous systems like buildbot, you shouldn't do any heavy
computation in the main event loop thread, those must be done in a thread
via deferToThread and co.

Those are the common issues you can have with performance
independantly from upgrade regressions:

1) Custom steps:
A lot of time, we see people struggling with performance when they just
have some custom step doing heavy computation that block the main thread
constantly, preventing all the very quick tasks to run in //.

2) too much logs
 In this case, there is not much to do beside reducing the log amount. This
would be the time to switch to a multi-master setup, where you put 2
masters for builds, and one master for web UI.
You can put those in the same machine/VM, no problem, the only work is to
have separate processes that each have several event queues. You can use
docker-compose or kubernetes in order to more easily create such
deployment. We don't have readily useable for that, but several people have
done and documented it, for example
https://github.com/pop/buildbot-on-kubernetes


I have developed the buildbot profiler in order to quickly find those. You
just have to install it as a plugin and start a profile whenever the
buildbot feels slow.
It is a statistical profiler, so it will not significantly change the
actual performance so it is safe to run in production.

https://pypi.org/project/buildbot-profiler/


Regards,
Pierre


Le mar. 12 janv. 2021 à 01:29, Yngve N. Pettersen <yngve at vivaldi.com> a
écrit :

> Hello all,
>
> We have just upgraded our buildbot system from 2.7 to 2.10.
>
> However, I am noticing performance issues when loading these pages:
>
>   Builds->Builders
>   Builds->Workers
>   individual builds
>
> Loading these can take several minutes, although there are periods of
> immediate responses.
>
> What I am seeing on the buildbot manager machine is that the Python3
> process hits 90-100% for the entire period.
>
> The Python version is 3.6.9 running on Ubuntu 18.04
>
> As far as I can tell, the Postgresql database is mostly idle during this
> period. I did do a full vacuum a few hours ago, in case that was the issue.
>
> There are about 40 builders, and 30 workers in the system, only about
> 10-15 of these have a 10-20 builds for the past few days, although most
> of
> these have active histories of 3000 builds (which do make me wonder if
> the
> problem could be a lack of limiting the DB queries, at present I have not
> inspected the DB queries).
>
> The individual builds can have very large log files in the build steps,
> in
> many cases tens of thousands of lines (we _are_ talking about a Chromium
> based project).
>
> Our changes in the builders and workers JS code are minimal (we are using
> a custom build of www-base), just using different information for the
> build labels (build version number), and grouping the builders, which
> should not be causing any performance issues. (we have larger changes in
> the individual builder view, where we include Git commit messages, and I
> have so far not seen any performance issues there)
>
> BTW: The line plots for build time and successes on builders seems to be
> MIA. Not sure if that is an upstream issue, or due to something in our
> www-base build.
>
> Do you have any suggestions for where to look for the cause of the problem?
>
>
> --
> Sincerely,
> Yngve N. Pettersen
> Vivaldi Technologies AS
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20210112/fd21e224/attachment.html>


More information about the users mailing list