[users at bb.net] buildbot CPU usage

[Pierre Tardy]

You can add following snippet to your master.cfg. It looks statprof does
not work on osx, so I cannot test it to the end, but I know similar did
work on my prod


import statprof
from  twisted.application.internet import TimerService
from buildbot.util.service import BuildbotService
class BuildbotTimerService(TimerService, BuildbotService):
    name = "timer"
    def __init__(self):
        BuildbotService.__init__(self)
        TimerService.__init__(self, 10, self.dump_stats)
    def dump_stats(self):
        statprof.stop()
        with open("profile.txt", "w") as f:
            statprof.display(f)
        statprof.start()

c['services'] = [ BuildbotTimerService() ]


Le mar. 9 août 2016 à 21:44, Francesco Di Mizio <francescodimizio at gmail.com>
a écrit :

> Pierre,
>
> if you can enlighten on how you intend to use statprof, I will be happy to
> give it a try.
> This problem here is on top of my list right now as it's making my prod
> env unusable.
>
> On Tue, Aug 9, 2016 at 8:25 PM, Pierre Tardy <tardyp at gmail.com> wrote:
>
>> Please note that cprofile is very intrusive and will lead to about 3x
>> slower code.
>>
>> So I would not run that in prod. This is why I prefer statprof which is
>> using a non intrusive method to get the stats
>>
>> Le mar. 9 août 2016 19:15, Francesco Di Mizio <francescodimizio at gmail.com>
>> a écrit :
>>
>>> On a side note I have managed to get twistd to dump a blob I can then
>>> load and analyze. Got to see if I can run this in production.
>>> If anybody is curious the following worked for me:
>>> twistd --savestats -n --profiler=cprofile
>>>  --profile=/crcdata/profile.stats -y ./buildbot.tac
>>>
>>>
>>>
>>>
>>> On Tue, Aug 9, 2016 at 5:21 PM, Francesco Di Mizio <
>>> francescodimizio at gmail.com> wrote:
>>>
>>>> Well that's expected to an extent - p4 poller is running 'p4 changes'
>>>> every pollinterval seconds.
>>>>
>>>> Anyway just as a test I've tried to disable the poller and still at
>>>> times I am seeing the CPU spiking up to above 100%. I believe I really need
>>>> to profile this somehow.
>>>>
>>>> On Tue, Aug 9, 2016 at 5:08 PM, Dan Kegel <dank at kegel.com> wrote:
>>>>
>>>>> Also watch 'top' and see if poller processes are hogging resources.
>>>>>
>>>>
>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20160809/120fd0fa/attachment.html>