[users at bb.net] buildbot CPU usage

Fri Aug 26 15:48:11 UTC 2016

I just went ahead and disabled the potentially too long logs and havent
seen a spice since then ;) finger crossed!

On Fri, Aug 26, 2016 at 5:35 PM, Pierre Tardy <tardyp at gmail.com> wrote:

> Hum,
> It might not be stuck, actually, but just spending very long time to
> compress the log.
>
> In theory, the log compression is not waited for, though.
>
> Note that if you stop a build that is waiting for a lost deferred, this
> will have no effect as you describe.
>
> You got absolutly no exception in twisted.log?
>
> Pierre
>
>
> Le ven. 26 août 2016 à 17:06, Neil Gilmore <ngilmore at grammatech.com> a
> écrit :
>
>> I'm currently looking at a step with 3 logs:
>> 81531 lines
>> 489285 lines
>> 489311 lines
>>
>> An earlier successful run would have that first log at 244080 lines.
>>
>> This particular build is stuck, though. :( (which is why I'm looking at
>> it.) (A bit off-topic, but I tried stopping it. The last step is marked
>> cancelled, but that's the only effect.)
>>
>>
>> Neil Gilmore
>> grammatech.com
>>
>>
>> On 8/26/2016 6:25 AM, Pierre Tardy wrote:
>>
>>
>> The ram looks like sufficient, it might be a good test to try and
>> increase the number of cpu for that VM.
>> In your trace, I can see the use of up to 7 threads at the same time, so
>> you might gain by going 8 CPUs
>>
>> Also, make sure that your VM host is not overbooked. In my experience of
>> using VMware VMs provided by IT, overbooking has been a source of
>> inexplicable performance issues.
>>
>> 12k lines is a lot, but buildbot shall support this kind of load without
>> issue.
>>
>> Le ven. 26 août 2016 à 12:05, Francesco Di Mizio <
>> francescodimizio at gmail.com> a écrit :
>>
>>> It's a vmware virtual machine with 4 GIGs RAM and 4 CPUs at 3Ghz. It runs,
>>> among other marginal things, 2 docker containers - one for the buildbot and
>>> one for the postgres db.
>>>
>>> The most beefy logs have around 12K lines. Is it too much?
>>> Also some other logs are read from the worker's filesystem and added as
>>> additional logs.
>>>
>>> On Fri, Aug 26, 2016 at 11:51 AM, Pierre Tardy <tardyp at gmail.com> wrote:
>>>
>>>> Cool!
>>>> I can indeed see 3 spikes.
>>>>
>>>> Looks related to logs and logs compression.
>>>>
>>>> What is the HW spec of your master machine?
>>>> How much log does your build generate?
>>>>
>>>> Pierre
>>>>
>>>> Le ven. 26 août 2016 à 11:42, Francesco Di Mizio <
>>>> francescodimizio at gmail.com> a écrit :
>>>>
>>>>> Pierre,
>>>>>
>>>>> I enabled it, waited 1 min and saw the spike, then stopped after a few
>>>>> secs. Attached the json.
>>>>> Awesome tool btw, work wonders!
>>>>>
>>>>> On Thu, Aug 25, 2016 at 1:14 PM, Pierre Tardy <tardyp at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> You can try to hit the button in the morning with a two hours gather
>>>>>> period, and hope that you see the spike during that period..
>>>>>>
>>>>>>
>>>>>> Le jeu. 25 août 2016 à 12:17, Francesco Di Mizio <
>>>>>> francescodimizio at gmail.com> a écrit :
>>>>>>
>>>>>>> Thanks a lot! Pierre I will def will give it a shot. I am not sure
>>>>>>> I'll be able to smash that 'start recording' button as the UI
>>>>>>> isusuallystuck when the CPU spikes. Updates to come!
>>>>>>>
>>>>>>> On Thu, Aug 25, 2016 at 10:45 AM, Pierre Tardy <tardyp at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Francesco,
>>>>>>>>
>>>>>>>> I spent some time in order to implement a profiler plugin for
>>>>>>>> buildbot
>>>>>>>>
>>>>>>>> You can give it a look, and send your profile.json file if you need
>>>>>>>> more analysis from me.
>>>>>>>> https://github.com/tardyp/buildbot_profiler
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Pierre
>>>>>>>>
>>>>>>>>
>>>>>>>> Le mer. 24 août 2016 à 22:43, Francesco Di Mizio <
>>>>>>>> francescodimizio at gmail.com> a écrit :
>>>>>>>>
>>>>>>>>> I've tried and it's not an easy task because of my Win into
>>>>>>>>> Vagrant into Docker setup.
>>>>>>>>> I'll try again soon when I get a Linux  box!
>>>>>>>>>
>>>>>>>>> On Fri, Aug 19, 2016 at 5:54 PM, Vasily <just.one.man at yandex.ru>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Umm, no. VTune has Python support starting 2017 Beta, and, well,
>>>>>>>>>> it was my team (at Intel) work actually :-)
>>>>>>>>>>
>>>>>>>>>> P.S. I'm from Intel, too.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Vasily
>>>>>>>>>> 19 авг. 2016 г. 18:17 пользователь "Francesco Di Mizio" <
>>>>>>>>>> francescodimizio at gmail.com> написал:
>>>>>>>>>>
>>>>>>>>>> I had thought you were making fun of Intel somehow ;)
>>>>>>>>>>>
>>>>>>>>>>> On Aug 19, 2016 5:07 PM, "Pierre Tardy" <tardyp at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> ahah
>>>>>>>>>>>
>>>>>>>>>>> I though this was a taunt on me being employed by Intel.
>>>>>>>>>>> I actually had mitigated experience with vtune few years ago,
>>>>>>>>>>> and didn't know they had python support until then.
>>>>>>>>>>> Being an opensource guy, I usually neglegate to look at
>>>>>>>>>>> proprietary stuff.
>>>>>>>>>>>
>>>>>>>>>>> Pierre
>>>>>>>>>>>
>>>>>>>>>>> Le ven. 19 août 2016 à 12:18, Vasily <just.one.man at yandex.ru> a
>>>>>>>>>>> écrit :
>>>>>>>>>>>
>>>>>>>>>>>> I'm again suggesting to look into Python profiling capabilities
>>>>>>>>>>>> of Intel® VTune™ Amplifier. It could run statistical profiling for a long
>>>>>>>>>>>> time and display CPU usage over time, so the developer can look at specific
>>>>>>>>>>>> time range where CPU usage was too high and see which functions were
>>>>>>>>>>>> executed.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Vasily
>>>>>>>>>>>> 19 авг. 2016 г. 11:57 пользователь "Pierre Tardy" <
>>>>>>>>>>>> tardyp at gmail.com> написал:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Francesco,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your described setup looks sane to me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The problems we are trying to catch are cpu spikes, as far as
>>>>>>>>>>>>> I understand, which does not happen for very long, but are very annoying
>>>>>>>>>>>>> for users, as it is blocking the reactor.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This problem is not easy to see in the profile you sent, as
>>>>>>>>>>>>> this profile is over long time, so we see the average of each method during
>>>>>>>>>>>>> the day and not the spikes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What would really be needed is a on-demand profiler which
>>>>>>>>>>>>> would detect cpu spikes and only log the stack traces during those times.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here is a nice blog pst explaining why statistic profiling is
>>>>>>>>>>>>> cool and easy to implement in python.
>>>>>>>>>>>>> https://nylas.com/blog/performance
>>>>>>>>>>>>>
>>>>>>>>>>>>> For 0.9.1 I want to concentrate on scalability, and write a
>>>>>>>>>>>>> debugging ui plugin based on those ideas (and probably code)
>>>>>>>>>>>>>
>>>>>>>>>>>>> That would be great if your team can help on that matter.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Pierre
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>>
>> _______________________________________________
>> users mailing listusers at buildbot.nethttps://lists.buildbot.net/mailman/listinfo/users
>>
>> _______________________________________________
>> users mailing list
>> users at buildbot.net
>> https://lists.buildbot.net/mailman/listinfo/users
>
>
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20160826/5d108ce9/attachment.html>