<div dir="ltr">Gave a try to the profiler option that comes with twistd<div><br></div><div>twistd --nodaemon --profile=statsObj --profiler=profile -y ./buildbot.tac<br></div><div><br></div><div>it does not seem to work. Not sure it's bbot or twisted itself.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 9, 2016 at 10:16 AM, Francesco Di Mizio <span dir="ltr"><<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>2016-08-08 23:55:01+0000 [-] P4 poll failed on atx-p4-buildproxy.rsi.global:<wbr>1666, //starcitizen/</div><div><span style="white-space:pre-wrap"> </span>Traceback (most recent call last):</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/buildbot/<wbr>changes/base.py", line 65, in doPoll</div><div><span style="white-space:pre-wrap"> </span> d = defer.maybeDeferred(self.poll)</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/twisted/<wbr>internet/defer.py", line 150, in maybeDeferred</div><div><span style="white-space:pre-wrap"> </span> result = f(*args, **kw)</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/buildbot/<wbr>changes/p4poller.py", line 162, in poll</div><div><span style="white-space:pre-wrap"> </span> d = self._poll()</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/twisted/<wbr>internet/defer.py", line 1274, in unwindGenerator</div><div><span style="white-space:pre-wrap"> </span> return _inlineCallbacks(None, gen, Deferred())</div><div><span style="white-space:pre-wrap"> </span>--- <exception caught here> ---</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/twisted/<wbr>internet/defer.py", line 1128, in _inlineCallbacks</div><div><span style="white-space:pre-wrap"> </span> result = g.send(result)</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/buildbot/<wbr>changes/p4poller.py", line 232, in _poll</div><div><span style="white-space:pre-wrap"> </span> result = yield self._get_process_output(args)</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/buildbot/<wbr>changes/p4poller.py", line 170, in _get_process_output</div><div><span style="white-space:pre-wrap"> </span> d = utils.getProcessOutput(self.<wbr>p4bin, args, env)</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/twisted/<wbr>internet/utils.py", line 128, in getProcessOutput</div><div><span style="white-space:pre-wrap"> </span> reactor)</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/twisted/<wbr>internet/utils.py", line 28, in _callProtocolWithDeferred</div><div><span style="white-space:pre-wrap"> </span> reactor.spawnProcess(p, executable, (executable,)+tuple(args), env, path)</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/twisted/<wbr>internet/posixbase.py", line 340, in spawnProcess</div><div><span style="white-space:pre-wrap"> </span> processProtocol, uid, gid, childFDs)</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/twisted/<wbr>internet/process.py", line 731, in __init__</div><div><span style="white-space:pre-wrap"> </span> self._fork(path, uid, gid, executable, args, environment, fdmap=fdmap)</div><div><span style="white-space:pre-wrap"> </span> File "/usr/local/lib/python2.7/<wbr>dist-packages/twisted/<wbr>internet/process.py", line 405, in _fork</div><div><span style="white-space:pre-wrap"> </span> self.pid = os.fork()</div><div><span style="white-space:pre-wrap"> </span>exceptions.OSError: [Errno 12] Cannot allocate memory</div><div><br></div><div>I'll run one day with the p4 poller disabled and see how it goes.</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 2, 2016 at 7:28 PM, Francesco Di Mizio <span dir="ltr"><<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Just one. Here is what the poller looks like<div><br></div><div><div>s = changes.P4Source(</div><div> p4port=config.p4_server,</div><div> p4user=config.p4_user,</div><div> p4passwd=config.p4_password,</div><div> p4base='//XXXXXXX/',</div><div> pollInterval=10,</div><div> pollAtLaunch = False,</div><div> split_file=lambda branchfile: branchfile.split('/',1),</div><div> encoding='cp437'</div><div>)</div><div><br></div><div><br><div><br></div><div><br></div></div></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 2, 2016 at 7:24 PM, Pierre Tardy <span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">How many projects are your pulling? I'll see if I can make a PoC of a builder which runs statprof</div><div><div><br><div class="gmail_quote"><div dir="ltr">Le mar. 2 août 2016 à 18:53, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the kind replies both of you.<div><br></div><div>@Pierre:</div><div><div>Not sure I get what you mean. Given the context, for a step to be CPU demanding it should be a master side step right? I happen to not have any.</div><div>What would you be profiling with statprof? </div><div>I'd really appreciate if you could elaborate on your idea.</div></div><div><br></div><div>Really all I can think of is the poller. I'll keep looking into it.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 2, 2016 at 6:36 PM, Dan Kegel <span dir="ltr"><<a href="mailto:dank@kegel.com" target="_blank">dank@kegel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">With gitpoller, it was easy to see; whenever the number of git<br>
sessions from the poller went over 0 or so, web gui performance was<br>
poor.<br>
And if it went over 10, well, you could kiss the gui goodbye for<br>
several minutes.<br>
<br>
One countermeasure was to randomize the polling intervals, a la<br>
<br>
interval=6 # minutes<br>
self['change_source'].append(<br>
# Fuzz the interval to avoid slamming the git server<br>
and hitting the MaxStartups or MaxSessions limits<br>
# If you hit them, twistd.log will have lots of<br>
"ssh_exchange_identification: Connection closed by remote host" errors<br>
# See <a href="http://trac.buildbot.net/ticket/2480" rel="noreferrer" target="_blank">http://trac.buildbot.net/ticke<wbr>t/2480</a><br>
changes.GitPoller(repourl, branches=branchnames,<br>
workdir='gitpoller-workdir-'+n<wbr>ame, pollinterval=interval*60 +<br>
random.uniform(-10, 10)))<br>
<br>
That made life just barely bearable, at least until number of projects<br>
polled was under 50 or so.<br>
What really helped was not using pollers anymore, and switching to<br>
gitlab's webhooks.<br>
We're at 190 now, of which 57 are still using gitpoller, and it's<br>
almost ok. (I really have<br>
to move the last 57 onto gitlab. Or, well, since they're not<br>
critical, increase the polling<br>
interval...)<br>
<div><div><br>
On Tue, Aug 2, 2016 at 9:13 AM, Pierre Tardy <<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>> wrote:<br>
> Hi,<br>
><br>
> Pollers are usually indeed not scaling as they, hmm, poll.<br>
> What you are describing here is hints that the twisted reactor thread is<br>
> always busy, which should not happen if you only start 10 builds.<br>
> You might have some custom steps which are doing something heavily cpu bound<br>
> in the main thread.<br>
> What I usually do is to use statprof:<br>
> <a href="https://pypi.python.org/pypi/statprof/" rel="noreferrer" target="_blank">https://pypi.python.org/pypi/s<wbr>tatprof/</a><br>
><br>
> in order to know what the cpu is doing.<br>
> You could create a builder which you can trig whenever you need, and which<br>
> would start the profiling, wait a few minutes, and then save profiling to a<br>
> file.<br>
><br>
><br>
><br>
> Le mar. 2 août 2016 à 17:53, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>><br>
> a écrit :<br>
>><br>
>> Hey Dan,<br>
>><br>
>> I am using a p4 poller. Maybe it's suffering from the same problems?<br>
>><br>
>> On Tue, Aug 2, 2016 at 5:45 PM, Francesco Di Mizio<br>
>> <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>> wrote:<br>
>>><br>
>>> I'd like to provide a bit more context.Right after restarting the master<br>
>>> and kicking off 10 builds CPU was at 110-120%. This made the UI unusable and<br>
>>> basically all the services were stuck, including the REST API.<br>
>>> After 3-4 minutes like this and WITH all the 10 builds still running the<br>
>>> CPU usage went down to 5%, stayed there for 5 minutes and all was smooth and<br>
>>> quick again. From then on it keps oscillating, I've seen spikes of 240% :(<br>
>>><br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> On Tue, Aug 2, 2016 at 4:12 PM, Francesco Di Mizio<br>
>>> <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>> wrote:<br>
>>>><br>
>>>> Sometimes it goes up to 140%. I was not able to relate this with a<br>
>>>> particular builds condition - seems like it can happen any time and is not<br>
>>>> related to how many builds are going on.<br>
>>>><br>
>>>> I usually realize the server got into this state because the web UI gets<br>
>>>> stuck. As soon as the CPU% goes back to normal values (2-3% most times) the<br>
>>>> web finishes loading just instantly.<br>
>>>><br>
>>>> Any pointers as to what might be causing this? Only reason I can think<br>
>>>> of is too many people trying to access the web UI simultaniously - may I be<br>
>>>> right?<br>
>>>><br>
>>><br>
>><br>
>> ______________________________<wbr>_________________<br>
>> users mailing list<br>
>> <a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a><br>
>> <a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mai<wbr>lman/listinfo/users</a><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> users mailing list<br>
> <a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a><br>
> <a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mai<wbr>lman/listinfo/users</a><br>
</div></div></blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>