[users at bb.net] buildbot CPU usage
Francesco Di Mizio
francescodimizio at gmail.com
Fri Aug 12 10:23:06 UTC 2016
It does work but statprof seems to stop getting samples. Last few lines
from it:
2016-08-12 10:19:43+0000 [-] 0.00 0.06 0.00
banana.py:167:gotItem
2016-08-12 10:19:43+0000 [-] 0.00 0.01 0.00
policies.py:718:setTimeout
2016-08-12 10:19:43+0000 [-] 0.00 0.01 0.00
encoder.py:434:_iterencode
2016-08-12 10:19:43+0000 [-] ---
2016-08-12 10:19:43+0000 [-] Sample count: 51
2016-08-12 10:19:43+0000 [-] Total time: 0.510000 seconds
The server is running.
On Thu, Aug 11, 2016 at 10:46 AM, Francesco Di Mizio <
francescodimizio at gmail.com> wrote:
> Pierre many thanks for that. I do not remember why the force scheduler api
> didnt suite my needs, that was my first approach.
> I will try the new snippet when I get to work in a few hours.
>
> On Wed, Aug 10, 2016 at 11:18 PM, Pierre Tardy <tardyp at gmail.com> wrote:
>
>> Hi,
>>
>> After carefully looking at your code, it does not look to have obvious
>> performance flaw.
>> I think it would have been cheaper for you to just use the forcescheduler
>> REST api, but if that code works for you this is totally fine.
>>
>> I saw that you did not manage to get sample in your prod as well, so I
>> did look at my code again, and now I am able to see the samples.
>>
>> statprof does not like to stop before start
>>
>> Please try again with this code:
>>
>>
>> import statprof
>> from twisted.application.internet import TimerService
>> from buildbot.util.service import BuildbotService
>> class BuildbotTimerService(TimerService, BuildbotService):
>> name = "timer"
>> def __init__(self):
>> BuildbotService.__init__(self)
>> TimerService.__init__(self, 10, self.dump_stats)
>> self.started = False
>> def dump_stats(self):
>> if not self.started:
>> statprof.start()
>> self.started = True
>> return
>> statprof.stop()
>> with open("profile.txt", "w") as f:
>> statprof.display(f)
>> statprof.display()
>> statprof.start()
>>
>> c['services'] = [ BuildbotTimerService() ]
>>
>>
>>
>> Le mer. 10 août 2016 à 14:14, Francesco Di Mizio <
>> francescodimizio at gmail.com> a écrit :
>>
>>> Many thanks. Find it attached.
>>>
>>> It's a tiny http server. People use a web page to send an HTTP request
>>> to it to kick off builds.
>>>
>>>
>>> On Wed, Aug 10, 2016 at 1:55 PM, Pierre Tardy <tardyp at gmail.com> wrote:
>>>
>>>> I'll take a look.
>>>>
>>>> Le mer. 10 août 2016 12:45, Francesco Di Mizio <
>>>> francescodimizio at gmail.com> a écrit :
>>>>
>>>>> I am starting to think it may be due to my own custom scheduler. If I
>>>>> uploaded the code somewhere could anybody take a look? it's a 150 lines py
>>>>> script.
>>>>>
>>>>> On Wed, Aug 10, 2016 at 12:10 PM, Francesco Di Mizio <
>>>>> francescodimizio at gmail.com> wrote:
>>>>>
>>>>>> Thanks for that Pierre. I have it on on production and the
>>>>>> performances do not degrade indeed.
>>>>>>
>>>>>> buildbot at e2123a50d1fb:~$ cat profile.txt
>>>>>> No samples recorded.
>>>>>>
>>>>>> I need to study statprof a bit now ;)
>>>>>>
>>>>>> On Wed, Aug 10, 2016 at 11:20 AM, Vasily <vassnlit at gmail.com> wrote:
>>>>>>
>>>>>>> One can also use Intel® VTune™, you know... :-)
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Vasily
>>>>>>> 09 авг. 2016 г. 23:56 пользователь "Pierre Tardy" <tardyp at gmail.com>
>>>>>>> написал:
>>>>>>>
>>>>>>>> You can add following snippet to your master.cfg. It looks statprof
>>>>>>>> does not work on osx, so I cannot test it to the end, but I know similar
>>>>>>>> did work on my prod
>>>>>>>>
>>>>>>>>
>>>>>>>> import statprof
>>>>>>>> from twisted.application.internet import TimerService
>>>>>>>> from buildbot.util.service import BuildbotService
>>>>>>>> class BuildbotTimerService(TimerService, BuildbotService):
>>>>>>>> name = "timer"
>>>>>>>> def __init__(self):
>>>>>>>> BuildbotService.__init__(self)
>>>>>>>> TimerService.__init__(self, 10, self.dump_stats)
>>>>>>>> def dump_stats(self):
>>>>>>>> statprof.stop()
>>>>>>>> with open("profile.txt", "w") as f:
>>>>>>>> statprof.display(f)
>>>>>>>> statprof.start()
>>>>>>>>
>>>>>>>> c['services'] = [ BuildbotTimerService() ]
>>>>>>>>
>>>>>>>>
>>>>>>>> Le mar. 9 août 2016 à 21:44, Francesco Di Mizio <
>>>>>>>> francescodimizio at gmail.com> a écrit :
>>>>>>>>
>>>>>>>>> Pierre,
>>>>>>>>>
>>>>>>>>> if you can enlighten on how you intend to use statprof, I will be
>>>>>>>>> happy to give it a try.
>>>>>>>>> This problem here is on top of my list right now as it's making my
>>>>>>>>> prod env unusable.
>>>>>>>>>
>>>>>>>>> On Tue, Aug 9, 2016 at 8:25 PM, Pierre Tardy <tardyp at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Please note that cprofile is very intrusive and will lead to
>>>>>>>>>> about 3x slower code.
>>>>>>>>>>
>>>>>>>>>> So I would not run that in prod. This is why I prefer statprof
>>>>>>>>>> which is using a non intrusive method to get the stats
>>>>>>>>>>
>>>>>>>>>> Le mar. 9 août 2016 19:15, Francesco Di Mizio <
>>>>>>>>>> francescodimizio at gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>>> On a side note I have managed to get twistd to dump a blob I can
>>>>>>>>>>> then load and analyze. Got to see if I can run this in production.
>>>>>>>>>>> If anybody is curious the following worked for me:
>>>>>>>>>>> twistd --savestats -n --profiler=cprofile --profile=/crcdata/
>>>>>>>>>>> profile.stats -y ./buildbot.tac
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 9, 2016 at 5:21 PM, Francesco Di Mizio <
>>>>>>>>>>> francescodimizio at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Well that's expected to an extent - p4 poller is running 'p4
>>>>>>>>>>>> changes' every pollinterval seconds.
>>>>>>>>>>>>
>>>>>>>>>>>> Anyway just as a test I've tried to disable the poller and
>>>>>>>>>>>> still at times I am seeing the CPU spiking up to above 100%. I believe I
>>>>>>>>>>>> really need to profile this somehow.
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 9, 2016 at 5:08 PM, Dan Kegel <dank at kegel.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Also watch 'top' and see if poller processes are hogging
>>>>>>>>>>>>> resources.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users at buildbot.net
>>>>>>>> https://lists.buildbot.net/mailman/listinfo/users
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20160812/3a93ec93/attachment.html>
More information about the users
mailing list