[users at bb.net] Performance issues in 2.10

[Yngve N. Pettersen]

Hello all,

We have just upgraded our buildbot system from 2.7 to 2.10.

However, I am noticing performance issues when loading these pages:

  Builds->Builders
  Builds->Workers
  individual builds

Loading these can take several minutes, although there are periods of  
immediate responses.

What I am seeing on the buildbot manager machine is that the Python3  
process hits 90-100% for the entire period.

The Python version is 3.6.9 running on Ubuntu 18.04

As far as I can tell, the Postgresql database is mostly idle during this  
period. I did do a full vacuum a few hours ago, in case that was the issue.

There are about 40 builders, and 30 workers in the system, only about  
10-15 of these have a 10-20 builds for the past few days, although most of  
these have active histories of 3000 builds (which do make me wonder if the  
problem could be a lack of limiting the DB queries, at present I have not  
inspected the DB queries).

The individual builds can have very large log files in the build steps, in  
many cases tens of thousands of lines (we _are_ talking about a Chromium  
based project).

Our changes in the builders and workers JS code are minimal (we are using  
a custom build of www-base), just using different information for the  
build labels (build version number), and grouping the builders, which  
should not be causing any performance issues. (we have larger changes in  
the individual builder view, where we include Git commit messages, and I  
have so far not seen any performance issues there)

BTW: The line plots for build time and successes on builders seems to be  
MIA. Not sure if that is an upstream issue, or due to something in our  
www-base build.

Do you have any suggestions for where to look for the cause of the problem?


-- 
Sincerely,
Yngve N. Pettersen
Vivaldi Technologies AS