[users at bb.net] Performance issues in 2.10

[Vlad Bogolin]

Hi,

So the changes issue I was referring to doesn't seem to be fixed (I checked
the latest Buildbot code) and looked back over my changes. The problem that
I identified is here
https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L320
because is some cases there would be an unmatched_filter or
unmatched_order. *So if this is indeed the case, you should see the warning
in the buildbot log file:*

"Warning: limited data api query is not backed by db because of following
filters..." as defined here
https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L322

There are two reasons why this happens. One, an incomplete definition of
the fieldMapping for the data/changes.py and this would be the fix:

+++ b/master/buildbot/data/changes.py
@@ -42,6 +42,19 @@ class FixerMixin:
         return change
     fieldMapping = {
         'changeid': 'changes.id',
+        'author': 'changes.author',
+        'committer': 'changes.committer',
+        'comments': 'changes.comments',
+        'branch': 'changes.branch',
+        'revision': 'changes.revision',
+        'revlink': 'changes.revlink',
+        'when_timestamp': 'changes.when_timestamp',
+        'category': 'changes.category',
+        'repository': 'changes.repository',
+        'codebase': 'changes.codebase',
+        'project': 'changes.project',
+        'sourcestampid': 'changes.sourcestampid',
+        'parent_changeids': 'changes.parent_changeids',
     }

and two not having all the columns in the select statement in order to be
able to check for matched/unmatched filtering or ordering as it is done
here
https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L269.
My solution was to get all the fields from the changes table from the
database (don't know if this is the best approach) as you can see here:

+++ b/master/buildbot/db/changes.py
@@ -249,7 +249,7 @@ class
ChangesConnectorComponent(base.DBConnectorComponent):
         def thd(conn):
             # get the changeids from the 'changes' table
             changes_tbl = self.db.model.changes
-            q = sa.select([changes_tbl.c.changeid])
+            q = changes_tbl.select()

However, since only the changeid would be needed, don't sure if this would
be the right approach. Also, I would suspect some missing tests because I
would expect this to be a pretty common use case.

Regarding the profiler, I also had some problems running it for a longer
time, so if you find a fix, please share it with me.

Cheers,
Vlad

On Wed, Jan 13, 2021 at 1:33 PM Pierre Tardy <tardyp at gmail.com> wrote:

>
> I am not sure why this wouldn't work. I vaguely recall there was an issue
> there, but can't figure it out staring at the code.
> As this is quite ancient, I am not sure the JS part will even build
> anymore :-/
>
> You can change the default values at that line in the python
>
> https://github.com/tardyp/buildbot_profiler/blob/master/buildbot_profiler/api.py#L193
> I think you will be able to force them by editing this file inside your
> virtualenv..
>
>
> Regards
> Pierre
>
>
> Le mer. 13 janv. 2021 à 11:46, Yngve N. Pettersen <yngve at vivaldi.com> a
> écrit :
>
>> > I insist on the buildbot profiler. What I was saying before is that you
>> > need to hit the record button before the problem appears, and put a
>> large
>> > enough record time to be sure to catch a spike.
>> > Then, you will be able to zoom to the cpu spike and catch the issue
>> > precisely.
>> >
>> > If the spike is in the order of minutes like you said, you can
>> configure
>> > it
>> > like this and get enough samples to get enough evidence to where the
>> code
>> > is actually spending time:
>> >
>> > ProfilerService(frequency=500, gatherperiod=60 * 60, mode='virtual',
>> > basepath=None, wantBuilds=100
>>
>> I tried configuring this with the settings dropdown in the WebGUI
>> plugin,
>> but AFAICT it is not working, it only gathers info for 30 seconds.
>>
>> I guess I must be holding it incorrectly.
>>
>> > This will record for one hour, and mitigate the memory used if you worry
>> > about it.
>>
>> --
>> Sincerely,
>> Yngve N. Pettersen
>> Vivaldi Technologies AS
>>
>

-- 
Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20210113/4b90e36b/attachment.html>