[users at bb.net] Performance issues in 2.10

[Pierre Tardy]

ohoh.
Indeed that is bad.

Yngve, do you have such messages in logs?

"Warning: limited data api query is not backed by db because of following
filters"

Pierre


Le mer. 13 janv. 2021 à 12:49, Vlad Bogolin <vlad at mariadb.org> a écrit :

> Hi,
>
> So the changes issue I was referring to doesn't seem to be fixed (I
> checked the latest Buildbot code) and looked back over my changes. The
> problem that I identified is here
> https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L320
> because is some cases there would be an unmatched_filter or
> unmatched_order. *So if this is indeed the case, you should see the
> warning in the buildbot log file:*
>
> "Warning: limited data api query is not backed by db because of following
> filters..." as defined here
> https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L322
>
> There are two reasons why this happens. One, an incomplete definition of
> the fieldMapping for the data/changes.py and this would be the fix:
>
> +++ b/master/buildbot/data/changes.py
> @@ -42,6 +42,19 @@ class FixerMixin:
>          return change
>      fieldMapping = {
>          'changeid': 'changes.id',
> +        'author': 'changes.author',
> +        'committer': 'changes.committer',
> +        'comments': 'changes.comments',
> +        'branch': 'changes.branch',
> +        'revision': 'changes.revision',
> +        'revlink': 'changes.revlink',
> +        'when_timestamp': 'changes.when_timestamp',
> +        'category': 'changes.category',
> +        'repository': 'changes.repository',
> +        'codebase': 'changes.codebase',
> +        'project': 'changes.project',
> +        'sourcestampid': 'changes.sourcestampid',
> +        'parent_changeids': 'changes.parent_changeids',
>      }
>
> and two not having all the columns in the select statement in order to be
> able to check for matched/unmatched filtering or ordering as it is done
> here
> https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L269.
> My solution was to get all the fields from the changes table from the
> database (don't know if this is the best approach) as you can see here:
>
> +++ b/master/buildbot/db/changes.py
> @@ -249,7 +249,7 @@ class
> ChangesConnectorComponent(base.DBConnectorComponent):
>          def thd(conn):
>              # get the changeids from the 'changes' table
>              changes_tbl = self.db.model.changes
> -            q = sa.select([changes_tbl.c.changeid])
> +            q = changes_tbl.select()
>
> However, since only the changeid would be needed, don't sure if this would
> be the right approach. Also, I would suspect some missing tests because I
> would expect this to be a pretty common use case.
>
> Regarding the profiler, I also had some problems running it for a longer
> time, so if you find a fix, please share it with me.
>
> Cheers,
> Vlad
>
> On Wed, Jan 13, 2021 at 1:33 PM Pierre Tardy <tardyp at gmail.com> wrote:
>
>>
>> I am not sure why this wouldn't work. I vaguely recall there was an issue
>> there, but can't figure it out staring at the code.
>> As this is quite ancient, I am not sure the JS part will even build
>> anymore :-/
>>
>> You can change the default values at that line in the python
>>
>> https://github.com/tardyp/buildbot_profiler/blob/master/buildbot_profiler/api.py#L193
>> I think you will be able to force them by editing this file inside your
>> virtualenv..
>>
>>
>> Regards
>> Pierre
>>
>>
>> Le mer. 13 janv. 2021 à 11:46, Yngve N. Pettersen <yngve at vivaldi.com> a
>> écrit :
>>
>>> > I insist on the buildbot profiler. What I was saying before is that you
>>> > need to hit the record button before the problem appears, and put a
>>> large
>>> > enough record time to be sure to catch a spike.
>>> > Then, you will be able to zoom to the cpu spike and catch the issue
>>> > precisely.
>>> >
>>> > If the spike is in the order of minutes like you said, you can
>>> configure
>>> > it
>>> > like this and get enough samples to get enough evidence to where the
>>> code
>>> > is actually spending time:
>>> >
>>> > ProfilerService(frequency=500, gatherperiod=60 * 60, mode='virtual',
>>> > basepath=None, wantBuilds=100
>>>
>>> I tried configuring this with the settings dropdown in the WebGUI
>>> plugin,
>>> but AFAICT it is not working, it only gathers info for 30 seconds.
>>>
>>> I guess I must be holding it incorrectly.
>>>
>>> > This will record for one hour, and mitigate the memory used if you
>>> worry
>>> > about it.
>>>
>>> --
>>> Sincerely,
>>> Yngve N. Pettersen
>>> Vivaldi Technologies AS
>>>
>>
>
> --
> Vlad
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20210113/005ee7b8/attachment.html>