[users at bb.net] slow reconfig with mysqld?

[Pedro Algarvio]

Don't really have a notion of the implications of not sorting, but I managed to reduce the CPU spike and reconfigure time by providing 'prioritizeBuilders` on the master config which just returns the unsorted builders.

Best,
Pedro.

On July 19, 2018 2:26:29 PM GMT+01:00, Dan Kegel <dank at kegel.com> wrote:
>About 1800 builders.   Cpu was high, mostly in mysqld, but buildbot
>was also busy:
>
>  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
>COMMAND
>26049 mysql     20   0 2404832 220032   6992 S 161.1  2.7 285:11.00
>mysqld
>25662 buildbot  20   0 4636132 3.019g  10932 S  24.9 38.7 202:59.14
>python3
>
>Normally it takes 2 minutes to reconfigure, so something odd was going
>on.
>
>buildbot sure takes a lot of RAM.
>
>On Thu, Jul 19, 2018 at 1:01 AM, Pedro Algarvio <pedro at algarvio.me>
>wrote:
>> How many builder are involved, more or less, in your setup?
>> Was your CPU pegged while reconfig was running?
>>
>> Best,
>> Pedro.
>>
>> On July 18, 2018 5:31:12 AM GMT+01:00, Dan Kegel <dank at kegel.com>
>wrote:
>>>
>>> Reran reconfig just now, and it took less than 2 minutes, as
>expected.
>>>
>>> Why the heck did it take an hour last time?
>>>
>>>
>>> On Tue, Jul 17, 2018 at 9:07 PM, Dan Kegel <dank at kegel.com> wrote:
>>>>
>>>>  Any guesses how long the reconfig took?
>>>>
>>>>  Scroll down to see:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>  2018-07-17 18:10:51-0700 [-] reconfig is ongoing for 3180 s
>>>>
>>>>  OMG.
>>>>
>>>>  On Tue, Jul 17, 2018 at 5:32 PM, Dan Kegel <dank at kegel.com> wrote:
>>>>>
>>>>>  Hi all,
>>>>>  I finally switched to mysqld instead of sqlite
>>>>>  when I made the leap to buildbot 1.x.
>>>>>  At first it was much faster.
>>>>>
>>>>>  Now, though.... 'buildbot reconfig' has been running for 900
>seconds
>>>>>  and still isn't done.  ps shows:
>>>>>
>>>>>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM    
>TIME+
>>>>>  COMMAND
>>>>>  26049 mysql     20   0 2404832 220032   6992 S 161.1  2.7
>285:11.00
>>>>>  mysqld
>>>>>  25662 buildbot  20   0 4636132 3.019g  10932 S  24.9 38.7
>202:59.14
>>>>>  python3
>>>>>
>>>>>  Seems like I've run into a gotcha.  Do I have to reframbulate the
>>>>>  primary cryptokey tv tables in mysqld or sommat?
>>>
>>> ________________________________
>>>
>>> users mailing list
>>> users at buildbot.net
>>> https://lists.buildbot.net/mailman/listinfo/users
>>
>>
>> Pedro Algarvio @ Phone

Pedro Algarvio @ Phone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20180719/445416cc/attachment.html>