[users at bb.net] slow reconfig with mysqld?
Dan Kegel
dank at kegel.com
Sat Jul 21 00:40:04 UTC 2018
That's interesting, thanks.
Next related problem: 'buildbot stop' is taking ages and ages. Here
it is, still going strong after 45 minutes:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
26049 mysql 20 0 2404832 159072 6644 S 146.5 1.9 1341:53
mysqld
25662 buildbot 20 0 5012164 3.484g 10560 S 3.3 44.7 860:05.55 python3
I have a clue this time:
mysql> SHOW PROCESSLIST;
+--------+----------+-----------------+-------+---------+------+------------------------------+------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State
| Info
|
+--------+----------+-----------------+-------+---------+------+------------------------------+------------------------------------------------------------------------------------------------------+
| 204924 | buildbot | localhost:49860 | bbmb4 | Sleep | 1621 |
| NULL
|
| 205225 | buildbot | localhost:51066 | bbmb4 | Sleep | 1062 |
| NULL
|
| 205432 | buildbot | localhost:51892 | bbmb4 | Sleep | 853 |
| NULL
|
| 210548 | buildbot | localhost:44126 | bbmb4 | Query | 0 |
Waiting for table level lock | INSERT INTO logchunks (logid,
first_line, last_line, content, compressed) VALUES (16233, 4534222, 45
|
| 210805 | buildbot | localhost:45152 | bbmb4 | Query | 1 |
Waiting for table level lock | INSERT INTO logchunks (logid,
first_line, last_line, content, compressed) VALUES (16234, 4489069, 44
|
| 211749 | buildbot | localhost:48930 | bbmb4 | Query | 0 |
Waiting for table level lock | INSERT INTO logchunks (logid,
first_line, last_line, content, compressed) VALUES (16224, 4576307, 45
|
| 211919 | buildbot | localhost:49612 | bbmb4 | Query | 1 |
Waiting for table level lock | SELECT logchunks.first_line,
logchunks.last_line, logchunks.content, logchunks.compressed
FROM logc |
| 211985 | buildbot | localhost:49878 | bbmb4 | Query | 1 |
Waiting for table level lock | INSERT INTO logchunks (logid,
first_line, last_line, content, compressed) VALUES (16216, 4436858, 44
|
| 211987 | buildbot | localhost:49886 | bbmb4 | Query | 1 |
Waiting for table level lock | INSERT INTO logchunks (logid,
first_line, last_line, content, compressed) VALUES (16240, 4250646, 42
|
| 211990 | buildbot | localhost:49898 | bbmb4 | Query | 1 |
updating | DELETE FROM logchunks WHERE
logchunks.logid = 16241 AND logchunks.first_line >= 4287220 AND
logchunk |
| 211997 | root | localhost | NULL | Query | 0 |
starting | SHOW PROCESSLIST
|
+--------+----------+-----------------+-------+---------+------+------------------------------+------------------------------------------------------------------------------------------------------+
11 rows in set (0.04 sec)
Evidently it's being overwhelmed by a build job with a chatty log!
If it is simply just too much being logged, is there a way to defend
against this (like after N lines, just stop recording)?
- Dan
On Thu, Jul 19, 2018 at 6:42 AM, Pedro Algarvio <pedro at algarvio.me> wrote:
> Don't really have a notion of the implications of not sorting, but I managed
> to reduce the CPU spike and reconfigure time by providing
> 'prioritizeBuilders` on the master config which just returns the unsorted
> builders.
>
> Best,
> Pedro.
>
>
> On July 19, 2018 2:26:29 PM GMT+01:00, Dan Kegel <dank at kegel.com> wrote:
>>
>> About 1800 builders. Cpu was high, mostly in mysqld, but buildbot
>> was also busy:
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 26049 mysql 20 0 2404832 220032 6992 S 161.1 2.7 285:11.00
>> mysqld
>> 25662 buildbot 20 0 4636132 3.019g 10932 S 24.9 38.7 202:59.14
>> python3
>>
>> Normally it takes 2 minutes to reconfigure, so something odd was going on.
>>
>> buildbot sure takes a lot of RAM.
>>
>> On Thu, Jul 19, 2018 at 1:01 AM, Pedro Algarvio <pedro at algarvio.me> wrote:
>>>
>>> How many builder are involved, more or less, in your setup?
>>> Was your CPU pegged while reconfig was running?
>>>
>>> Best,
>>> Pedro.
>>>
>>> On July 18, 2018 5:31:12 AM GMT+01:00, Dan Kegel <dank at kegel.com> wrote:
>>>>
>>>>
>>>> Reran reconfig just now, and it took less than 2 minutes, as expected.
>>>>
>>>> Why the heck did it take an hour last time?
>>>>
>>>>
>>>> On Tue, Jul 17, 2018 at 9:07 PM, Dan Kegel <dank at kegel.com> wrote:
>>>>>
>>>>>
>>>>> Any guesses how long the reconfig took?
>>>>>
>>>>> Scroll down to see:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2018-07-17 18:10:51-0700 [-] reconfig is ongoing for 3180 s
>>>>>
>>>>> OMG.
>>>>>
>>>>> On Tue, Jul 17, 2018 at 5:32 PM, Dan Kegel <dank at kegel.com> wrote:
>>>>>>
>>>>>>
>>>>>> Hi all,
>>>>>> I finally switched to mysqld instead of sqlite
>>>>>> when I made the leap to buildbot 1.x.
>>>>>> At first it was much faster.
>>>>>>
>>>>>> Now, though.... 'buildbot reconfig' has been running for 900 seconds
>>>>>> and still isn't done. ps shows:
>>>>>>
>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>>>>>> COMMAND
>>>>>> 26049 mysql 20 0 2404832 220032 6992 S 161.1 2.7 285:11.00
>>>>>> mysqld
>>>>>> 25662 buildbot 20 0 4636132 3.019g 10932 S 24.9 38.7 202:59.14
>>>>>> python3
>>>>>>
>>>>>> Seems like I've run into a gotcha. Do I have to reframbulate the
>>>>>> primary cryptokey tv tables in mysqld or sommat?
>>>>
>>>>
>>>> ________________________________
>>>>
>>>>
>>>> users mailing list
>>>> users at buildbot.net
>>>> https://lists.buildbot.net/mailman/listinfo/users
>>>
>>>
>>>
>>> Pedro Algarvio @ Phone
>
>
> Pedro Algarvio @ Phone
More information about the users
mailing list