<div dir="ltr">Hy Francesco,<div><br></div><div>This is weird, as on my environment I can see the sample count growing. It only grows if the process is not idle, though</div><div>So I have to use the master ui to actually make the profile change on my test environment.</div><div>Your profile looks like fairly idle. I can see pprint and write to logfile which might be not scalable if the data start growing do you have some heavy debug log in your prod?</div><div><br></div><div>Here is an updated more simple snippet, but this should not change many thing.</div><div>You can improve the TimerService '10' value to something bigger in order to dump less often. </div><div>Are you sure you did reproduce the slowness while profiling?</div><div><br></div><div><br></div><div><div>import statprof</div><div>from twisted.application.internet import TimerService</div><div>from buildbot.util.service import BuildbotService</div><div>from twisted.internet import reactor</div><div><br></div><div>class BuildbotTimerService(TimerService, BuildbotService):</div><div> name = "timer"</div><div> def __init__(self):</div><div> BuildbotService.__init__(self)</div><div> TimerService.__init__(self, 10, self.dump_stats)</div><div> reactor.callFromThread(statprof.start)</div><div><br></div><div> def dump_stats(self):</div><div> statprof.stop()</div><div> with open("profile.txt", "w") as f:</div><div> statprof.display(f)</div><div> statprof.display()</div><div> statprof.start()</div><div><br></div><div>c['services'] = [ BuildbotTimerService() ]</div></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">Le ven. 12 août 2016 à 14:21, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com">francescodimizio@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sure thing. it's attached. <div>Please note the sampe count has been stuck at 51 for over 1 hour even if the master is running.<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 12, 2016 at 12:38 PM, Pierre Tardy <span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">The first lines are the important ones. Can you send the full file?</p><div><div>
<br><div class="gmail_quote"><div dir="ltr">Le ven. 12 août 2016 12:23, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It does work but statprof seems to stop getting samples. Last few lines from it:<div><br></div><div><div>2016-08-12 10:19:43+0000 [-] 0.00 0.06 0.00 banana.py:167:gotItem</div><div>2016-08-12 10:19:43+0000 [-] 0.00 0.01 0.00 policies.py:718:setTimeout</div><div>2016-08-12 10:19:43+0000 [-] 0.00 0.01 0.00 encoder.py:434:_iterencode</div><div>2016-08-12 10:19:43+0000 [-] ---</div><div>2016-08-12 10:19:43+0000 [-] Sample count: 51</div><div>2016-08-12 10:19:43+0000 [-] Total time: 0.510000 seconds</div></div><div><br></div><div>The server is running.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 11, 2016 at 10:46 AM, Francesco Di Mizio <span dir="ltr"><<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Pierre many thanks for that. I do not remember why the force scheduler api didnt suite my needs, that was my first approach.<div>I will try the new snippet when I get to work in a few hours.</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 10, 2016 at 11:18 PM, Pierre Tardy <span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>After carefully looking at your code, it does not look to have obvious performance flaw.</div><div>I think it would have been cheaper for you to just use the forcescheduler REST api, but if that code works for you this is totally fine.</div><div><br></div><div>I saw that you did not manage to get sample in your prod as well, so I did look at my code again, and now I am able to see the samples.</div><div><br></div><div>statprof does not like to stop before start</div><div><br></div><div>Please try again with this code:</div><div><br></div><div><span><div><br></div><div>import statprof</div><div>from twisted.application.internet import TimerService</div><div>from buildbot.util.service import BuildbotService</div><div>class BuildbotTimerService(TimerService, BuildbotService):</div><div> name = "timer"</div><div> def __init__(self):</div><div> BuildbotService.__init__(self)</div><div> TimerService.__init__(self, 10, self.dump_stats)</div></span><div> self.started = False</div><div> def dump_stats(self):</div><div> if not self.started:</div><div> statprof.start()</div><div> self.started = True</div><div> return</div><span><div> statprof.stop()</div><div> with open("profile.txt", "w") as f:</div><div> statprof.display(f)</div></span><div> statprof.display()</div><span><div> statprof.start()</div><div><br></div><div>c['services'] = [ BuildbotTimerService() ]</div></span></div><div><br></div><div><br></div></div><div><div><br><div class="gmail_quote"><div dir="ltr">Le mer. 10 août 2016 à 14:14, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Many thanks. Find it attached. <div><br></div><div>It's a tiny http server. People use a web page to send an HTTP request to it to kick off builds.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 10, 2016 at 1:55 PM, Pierre Tardy <span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">I'll take a look. </p><div><div>
<br><div class="gmail_quote"><div dir="ltr">Le mer. 10 août 2016 12:45, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I am starting to think it may be due to my own custom scheduler. If I uploaded the code somewhere could anybody take a look? it's a 150 lines py script.</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 10, 2016 at 12:10 PM, Francesco Di Mizio <span dir="ltr"><<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for that Pierre. I have it on on production and the performances do not degrade indeed.<div><br></div><div>buildbot@e2123a50d1fb:~$ cat profile.txt<br><div>No samples recorded.<br></div></div><div><br></div><div>I need to study statprof a bit now ;)</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 10, 2016 at 11:20 AM, Vasily <span dir="ltr"><<a href="mailto:vassnlit@gmail.com" target="_blank">vassnlit@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">One can also use Intel® VTune™, you know... :-)</p>
<p dir="ltr">Thanks, <br>
Vasily</p>
<div class="gmail_quote">09 авг. 2016 г. 23:56 пользователь "Pierre Tardy" <<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>> написал:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div>You can add following snippet to your master.cfg. It looks statprof does not work on osx, so I cannot test it to the end, but I know similar did work on my prod</div><div><br></div><div><br></div><div>import statprof</div><div>from twisted.application.internet import TimerService</div><div>from buildbot.util.service import BuildbotService</div><div>class BuildbotTimerService(TimerService, BuildbotService):</div><div> name = "timer"</div><div> def __init__(self):</div><div> BuildbotService.__init__(self)</div><div> TimerService.__init__(self, 10, self.dump_stats)</div><div> def dump_stats(self):</div><div> <span style="line-height:1.5"> statprof.stop()</span></div><div> with open("profile.txt", "w") as f:</div><div> statprof.display(f)</div><div> <span style="line-height:1.5"> statprof.start()</span></div><div><br></div><div>c['services'] = [ BuildbotTimerService() ]</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">Le mar. 9 août 2016 à 21:44, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Pierre,<div><br></div><div>if you can enlighten on how you intend to use statprof, I will be happy to give it a try.</div><div>This problem here is on top of my list right now as it's making my prod env unusable.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 9, 2016 at 8:25 PM, Pierre Tardy <span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Please note that cprofile is very intrusive and will lead to about 3x slower code.</p>
<p dir="ltr"> So I would not run that in prod. This is why I prefer statprof which is using a non intrusive method to get the stats</p><div><div>
<br><div class="gmail_quote"><div dir="ltr">Le mar. 9 août 2016 19:15, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">On a side note I have managed to get twistd to dump a blob I can then load and analyze. Got to see if I can run this in production.<div>If anybody is curious the following worked for me:</div><div>twistd --savestats -n --profiler=cprofile --profile=/crcdata/<a href="http://profile.st" target="_blank">profile.st</a>ats -y ./buildbot.tac</div><div><br></div><div><br><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 9, 2016 at 5:21 PM, Francesco Di Mizio <span dir="ltr"><<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Well that's expected to an extent - p4 poller is running 'p4 changes' every pollinterval seconds.<div><br></div><div>Anyway just as a test I've tried to disable the poller and still at times I am seeing the CPU spiking up to above 100%. I believe I really need to profile this somehow.</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 9, 2016 at 5:08 PM, Dan Kegel <span dir="ltr"><<a href="mailto:dank@kegel.com" target="_blank">dank@kegel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Also watch 'top' and see if poller processes are hogging resources.</p>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
</blockquote></div>
<br></div></div><span>_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a><br>
<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br></span></blockquote></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
</blockquote></div>