[users at bb.net] buildbot CPU usage

[Pierre Tardy]

Buildbot <0.9.3 did compress the whole log at once inside a transaction. I
can imagine this taking very long with 100% CPU.


Le mer. 1 févr. 2017 18:12, Francesco Di Mizio <francescodimizio at gmail.com>
a écrit :

> Hey Pierre,
>
> what I noticed is that the query does not take long at all when I run it
> myself.
> When it is issued through buildbot the state though, it is in a 'idle in
> transaction' state, which seems to be something bad according to many blogs
> on the web. As if bbot is waiting for postgres to do something but postgres
> isnt actually doing anything.
>
> Will give 9.3 but at this point I doubt it'll help.
>
> Thanks!
>
>
> On Wed, Feb 1, 2017 at 4:56 PM, Pierre Tardy <tardyp at gmail.com> wrote:
>
>
>
> Le lun. 30 janv. 2017 à 17:49, Francesco Di Mizio <
> francescodimizio at gmail.com> a écrit :
>
> Digging this up again. Havent moved to a physical server yet. I will first
> be splitting the bbot server and postgres on different hosts.
>
> What I've noticed the CPU spiking is always (and I rea) due to queries
> like:
> SELECT logchunks.first_line, logchunks.last_line, logchunks.content,
> logchunks.compressed FROM logchunks WHERE logchunks.logid = 56343 ORDER BY
> logchunks.first_line
>
> If I am correct that call can only come from CompressLog
> <https://github.com/buildbot/buildbot/blob/cfe60e31206748bcf01232eb13c0d6177cd04d5d/master/buildbot/db/logs.py#L252> .
> They do take a very long time (even 1 minute). While running all the system
> is stuck. I wonder if I should use some custom postgres config, I am simply
> running the official postgres 9.5 container.
>
> Hi Francesco,
> If your logs are not anormally large, like you said already this should
> not take one minute.
>
> That said, I have released in 0.9.3 an optimization that removes that
> particular request so that the compression happens on a stream of smaller
> requests.
>
> You might consider an upgrade in order to see if the situation improves.
>
> Pierre
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20170203/4efbef5d/attachment.html>