[users at bb.net] buildbot CPU usage

[Francesco Di Mizio]

And I believe even a dedicated quick HHD wouldnt hurt to access Postgres.
Will ask IT to make these long due changes and then update this thread.
As always, thanks a lot.

On Fri, Aug 26, 2016 at 1:25 PM, Pierre Tardy <tardyp at gmail.com> wrote:

>
> The ram looks like sufficient, it might be a good test to try and increase
> the number of cpu for that VM.
> In your trace, I can see the use of up to 7 threads at the same time, so
> you might gain by going 8 CPUs
>
> Also, make sure that your VM host is not overbooked. In my experience of
> using VMware VMs provided by IT, overbooking has been a source of
> inexplicable performance issues.
>
> 12k lines is a lot, but buildbot shall support this kind of load without
> issue.
>
> Le ven. 26 août 2016 à 12:05, Francesco Di Mizio <
> francescodimizio at gmail.com> a écrit :
>
>> It's a vmware virtual machine with 4 GIGs RAM and 4 CPUs at 3Ghz. It runs,
>> among other marginal things, 2 docker containers - one for the buildbot and
>> one for the postgres db.
>>
>> The most beefy logs have around 12K lines. Is it too much?
>> Also some other logs are read from the worker's filesystem and added as
>> additional logs.
>>
>> On Fri, Aug 26, 2016 at 11:51 AM, Pierre Tardy <tardyp at gmail.com> wrote:
>>
>>> Cool!
>>> I can indeed see 3 spikes.
>>>
>>> Looks related to logs and logs compression.
>>>
>>> What is the HW spec of your master machine?
>>> How much log does your build generate?
>>>
>>> Pierre
>>>
>>> Le ven. 26 août 2016 à 11:42, Francesco Di Mizio <
>>> francescodimizio at gmail.com> a écrit :
>>>
>>>> Pierre,
>>>>
>>>> I enabled it, waited 1 min and saw the spike, then stopped after a few
>>>> secs. Attached the json.
>>>> Awesome tool btw, work wonders!
>>>>
>>>> On Thu, Aug 25, 2016 at 1:14 PM, Pierre Tardy <tardyp at gmail.com> wrote:
>>>>
>>>>> You can try to hit the button in the morning with a two hours gather
>>>>> period, and hope that you see the spike during that period..
>>>>>
>>>>>
>>>>> Le jeu. 25 août 2016 à 12:17, Francesco Di Mizio <
>>>>> francescodimizio at gmail.com> a écrit :
>>>>>
>>>>>> Thanks a lot! Pierre I will def will give it a shot. I am not sure
>>>>>> I'll be able to smash that 'start recording' button as the UI
>>>>>> isusuallystuck when the CPU spikes. Updates to come!
>>>>>>
>>>>>> On Thu, Aug 25, 2016 at 10:45 AM, Pierre Tardy <tardyp at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Francesco,
>>>>>>>
>>>>>>> I spent some time in order to implement a profiler plugin for
>>>>>>> buildbot
>>>>>>>
>>>>>>> You can give it a look, and send your profile.json file if you need
>>>>>>> more analysis from me.
>>>>>>> https://github.com/tardyp/buildbot_profiler
>>>>>>>
>>>>>>> Regards,
>>>>>>> Pierre
>>>>>>>
>>>>>>>
>>>>>>> Le mer. 24 août 2016 à 22:43, Francesco Di Mizio <
>>>>>>> francescodimizio at gmail.com> a écrit :
>>>>>>>
>>>>>>>> I've tried and it's not an easy task because of my Win into Vagrant
>>>>>>>> into Docker setup.
>>>>>>>> I'll try again soon when I get a Linux  box!
>>>>>>>>
>>>>>>>> On Fri, Aug 19, 2016 at 5:54 PM, Vasily <just.one.man at yandex.ru>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Umm, no. VTune has Python support starting 2017 Beta, and, well,
>>>>>>>>> it was my team (at Intel) work actually :-)
>>>>>>>>>
>>>>>>>>> P.S. I'm from Intel, too.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Vasily
>>>>>>>>> 19 авг. 2016 г. 18:17 пользователь "Francesco Di Mizio" <
>>>>>>>>> francescodimizio at gmail.com> написал:
>>>>>>>>>
>>>>>>>>> I had thought you were making fun of Intel somehow ;)
>>>>>>>>>>
>>>>>>>>>> On Aug 19, 2016 5:07 PM, "Pierre Tardy" <tardyp at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> ahah
>>>>>>>>>>
>>>>>>>>>> I though this was a taunt on me being employed by Intel.
>>>>>>>>>> I actually had mitigated experience with vtune few years ago, and
>>>>>>>>>> didn't know they had python support until then.
>>>>>>>>>> Being an opensource guy, I usually neglegate to look at
>>>>>>>>>> proprietary stuff.
>>>>>>>>>>
>>>>>>>>>> Pierre
>>>>>>>>>>
>>>>>>>>>> Le ven. 19 août 2016 à 12:18, Vasily <just.one.man at yandex.ru> a
>>>>>>>>>> écrit :
>>>>>>>>>>
>>>>>>>>>>> I'm again suggesting to look into Python profiling capabilities
>>>>>>>>>>> of Intel® VTune™ Amplifier. It could run statistical profiling for a long
>>>>>>>>>>> time and display CPU usage over time, so the developer can look at specific
>>>>>>>>>>> time range where CPU usage was too high and see which functions were
>>>>>>>>>>> executed.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Vasily
>>>>>>>>>>> 19 авг. 2016 г. 11:57 пользователь "Pierre Tardy" <
>>>>>>>>>>> tardyp at gmail.com> написал:
>>>>>>>>>>>
>>>>>>>>>>> Hi Francesco,
>>>>>>>>>>>>
>>>>>>>>>>>> Your described setup looks sane to me.
>>>>>>>>>>>>
>>>>>>>>>>>> The problems we are trying to catch are cpu spikes, as far as I
>>>>>>>>>>>> understand, which does not happen for very long, but are very annoying for
>>>>>>>>>>>> users, as it is blocking the reactor.
>>>>>>>>>>>>
>>>>>>>>>>>> This problem is not easy to see in the profile you sent, as
>>>>>>>>>>>> this profile is over long time, so we see the average of each method during
>>>>>>>>>>>> the day and not the spikes.
>>>>>>>>>>>>
>>>>>>>>>>>> What would really be needed is a on-demand profiler which would
>>>>>>>>>>>> detect cpu spikes and only log the stack traces during those times.
>>>>>>>>>>>>
>>>>>>>>>>>> Here is a nice blog pst explaining why statistic profiling is
>>>>>>>>>>>> cool and easy to implement in python.
>>>>>>>>>>>> https://nylas.com/blog/performance
>>>>>>>>>>>>
>>>>>>>>>>>> For 0.9.1 I want to concentrate on scalability, and write a
>>>>>>>>>>>> debugging ui plugin based on those ideas (and probably code)
>>>>>>>>>>>>
>>>>>>>>>>>> That would be great if your team can help on that matter.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Pierre
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20160826/19adb627/attachment.html>