[users at bb.net] buildbot CPU usage

Fri Aug 26 11:25:03 UTC 2016

The ram looks like sufficient, it might be a good test to try and increase
the number of cpu for that VM.
In your trace, I can see the use of up to 7 threads at the same time, so
you might gain by going 8 CPUs

Also, make sure that your VM host is not overbooked. In my experience of
using VMware VMs provided by IT, overbooking has been a source of
inexplicable performance issues.

12k lines is a lot, but buildbot shall support this kind of load without
issue.

Le ven. 26 août 2016 à 12:05, Francesco Di Mizio <francescodimizio at gmail.com>
a écrit :

> It's a vmware virtual machine with 4 GIGs RAM and 4 CPUs at 3Ghz. It runs,
> among other marginal things, 2 docker containers - one for the buildbot and
> one for the postgres db.
>
> The most beefy logs have around 12K lines. Is it too much?
> Also some other logs are read from the worker's filesystem and added as
> additional logs.
>
> On Fri, Aug 26, 2016 at 11:51 AM, Pierre Tardy <tardyp at gmail.com> wrote:
>
>> Cool!
>> I can indeed see 3 spikes.
>>
>> Looks related to logs and logs compression.
>>
>> What is the HW spec of your master machine?
>> How much log does your build generate?
>>
>> Pierre
>>
>> Le ven. 26 août 2016 à 11:42, Francesco Di Mizio <
>> francescodimizio at gmail.com> a écrit :
>>
>>> Pierre,
>>>
>>> I enabled it, waited 1 min and saw the spike, then stopped after a few
>>> secs. Attached the json.
>>> Awesome tool btw, work wonders!
>>>
>>> On Thu, Aug 25, 2016 at 1:14 PM, Pierre Tardy <tardyp at gmail.com> wrote:
>>>
>>>> You can try to hit the button in the morning with a two hours gather
>>>> period, and hope that you see the spike during that period..
>>>>
>>>>
>>>> Le jeu. 25 août 2016 à 12:17, Francesco Di Mizio <
>>>> francescodimizio at gmail.com> a écrit :
>>>>
>>>>> Thanks a lot! Pierre I will def will give it a shot. I am not sure
>>>>> I'll be able to smash that 'start recording' button as the UI
>>>>> isusuallystuck when the CPU spikes. Updates to come!
>>>>>
>>>>> On Thu, Aug 25, 2016 at 10:45 AM, Pierre Tardy <tardyp at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Francesco,
>>>>>>
>>>>>> I spent some time in order to implement a profiler plugin for buildbot
>>>>>>
>>>>>> You can give it a look, and send your profile.json file if you need
>>>>>> more analysis from me.
>>>>>> https://github.com/tardyp/buildbot_profiler
>>>>>>
>>>>>> Regards,
>>>>>> Pierre
>>>>>>
>>>>>>
>>>>>> Le mer. 24 août 2016 à 22:43, Francesco Di Mizio <
>>>>>> francescodimizio at gmail.com> a écrit :
>>>>>>
>>>>>>> I've tried and it's not an easy task because of my Win into Vagrant
>>>>>>> into Docker setup.
>>>>>>> I'll try again soon when I get a Linux  box!
>>>>>>>
>>>>>>> On Fri, Aug 19, 2016 at 5:54 PM, Vasily <just.one.man at yandex.ru>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Umm, no. VTune has Python support starting 2017 Beta, and, well, it
>>>>>>>> was my team (at Intel) work actually :-)
>>>>>>>>
>>>>>>>> P.S. I'm from Intel, too.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Vasily
>>>>>>>> 19 авг. 2016 г. 18:17 пользователь "Francesco Di Mizio" <
>>>>>>>> francescodimizio at gmail.com> написал:
>>>>>>>>
>>>>>>>> I had thought you were making fun of Intel somehow ;)
>>>>>>>>>
>>>>>>>>> On Aug 19, 2016 5:07 PM, "Pierre Tardy" <tardyp at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> ahah
>>>>>>>>>
>>>>>>>>> I though this was a taunt on me being employed by Intel.
>>>>>>>>> I actually had mitigated experience with vtune few years ago, and
>>>>>>>>> didn't know they had python support until then.
>>>>>>>>> Being an opensource guy, I usually neglegate to look at
>>>>>>>>> proprietary stuff.
>>>>>>>>>
>>>>>>>>> Pierre
>>>>>>>>>
>>>>>>>>> Le ven. 19 août 2016 à 12:18, Vasily <just.one.man at yandex.ru> a
>>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>>> I'm again suggesting to look into Python profiling capabilities
>>>>>>>>>> of Intel® VTune™ Amplifier. It could run statistical profiling for a long
>>>>>>>>>> time and display CPU usage over time, so the developer can look at specific
>>>>>>>>>> time range where CPU usage was too high and see which functions were
>>>>>>>>>> executed.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Vasily
>>>>>>>>>> 19 авг. 2016 г. 11:57 пользователь "Pierre Tardy" <
>>>>>>>>>> tardyp at gmail.com> написал:
>>>>>>>>>>
>>>>>>>>>> Hi Francesco,
>>>>>>>>>>>
>>>>>>>>>>> Your described setup looks sane to me.
>>>>>>>>>>>
>>>>>>>>>>> The problems we are trying to catch are cpu spikes, as far as I
>>>>>>>>>>> understand, which does not happen for very long, but are very annoying for
>>>>>>>>>>> users, as it is blocking the reactor.
>>>>>>>>>>>
>>>>>>>>>>> This problem is not easy to see in the profile you sent, as this
>>>>>>>>>>> profile is over long time, so we see the average of each method during the
>>>>>>>>>>> day and not the spikes.
>>>>>>>>>>>
>>>>>>>>>>> What would really be needed is a on-demand profiler which would
>>>>>>>>>>> detect cpu spikes and only log the stack traces during those times.
>>>>>>>>>>>
>>>>>>>>>>> Here is a nice blog pst explaining why statistic profiling is
>>>>>>>>>>> cool and easy to implement in python.
>>>>>>>>>>> https://nylas.com/blog/performance
>>>>>>>>>>>
>>>>>>>>>>> For 0.9.1 I want to concentrate on scalability, and write a
>>>>>>>>>>> debugging ui plugin based on those ideas (and probably code)
>>>>>>>>>>>
>>>>>>>>>>> That would be great if your team can help on that matter.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Pierre
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20160826/f27f930e/attachment.html>