[users at bb.net] slow reconfig with mysqld?

[Dan Kegel]

About 1800 builders.   Cpu was high, mostly in mysqld, but buildbot
was also busy:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
COMMAND
26049 mysql     20   0 2404832 220032   6992 S 161.1  2.7 285:11.00
mysqld
25662 buildbot  20   0 4636132 3.019g  10932 S  24.9 38.7 202:59.14
python3

Normally it takes 2 minutes to reconfigure, so something odd was going on.

buildbot sure takes a lot of RAM.

On Thu, Jul 19, 2018 at 1:01 AM, Pedro Algarvio <pedro at algarvio.me> wrote:
> How many builder are involved, more or less, in your setup?
> Was your CPU pegged while reconfig was running?
>
> Best,
> Pedro.
>
> On July 18, 2018 5:31:12 AM GMT+01:00, Dan Kegel <dank at kegel.com> wrote:
>>
>> Reran reconfig just now, and it took less than 2 minutes, as expected.
>>
>> Why the heck did it take an hour last time?
>>
>>
>> On Tue, Jul 17, 2018 at 9:07 PM, Dan Kegel <dank at kegel.com> wrote:
>>>
>>>  Any guesses how long the reconfig took?
>>>
>>>  Scroll down to see:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>  2018-07-17 18:10:51-0700 [-] reconfig is ongoing for 3180 s
>>>
>>>  OMG.
>>>
>>>  On Tue, Jul 17, 2018 at 5:32 PM, Dan Kegel <dank at kegel.com> wrote:
>>>>
>>>>  Hi all,
>>>>  I finally switched to mysqld instead of sqlite
>>>>  when I made the leap to buildbot 1.x.
>>>>  At first it was much faster.
>>>>
>>>>  Now, though.... 'buildbot reconfig' has been running for 900 seconds
>>>>  and still isn't done.  ps shows:
>>>>
>>>>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
>>>>  COMMAND
>>>>  26049 mysql     20   0 2404832 220032   6992 S 161.1  2.7 285:11.00
>>>>  mysqld
>>>>  25662 buildbot  20   0 4636132 3.019g  10932 S  24.9 38.7 202:59.14
>>>>  python3
>>>>
>>>>  Seems like I've run into a gotcha.  Do I have to reframbulate the
>>>>  primary cryptokey tv tables in mysqld or sommat?
>>
>> ________________________________
>>
>> users mailing list
>> users at buildbot.net
>> https://lists.buildbot.net/mailman/listinfo/users
>
>
> Pedro Algarvio @ Phone