<div dir="ltr">How many projects are your pulling? I'll see if I can make a PoC of a builder which runs statprof</div><br><div class="gmail_quote"><div dir="ltr">Le mar. 2 août 2016 à 18:53, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com">francescodimizio@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the kind replies both of you.<div><br></div><div>@Pierre:</div><div><div>Not sure I get what you mean. Given the context, for a step to be CPU demanding it should be a master side step right? I happen to not have any.</div><div>What would you be profiling with statprof? </div><div>I'd really appreciate if you could elaborate on your idea.</div></div><div><br></div><div>Really all I can think of is the poller. I'll keep looking into it.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 2, 2016 at 6:36 PM, Dan Kegel <span dir="ltr"><<a href="mailto:dank@kegel.com" target="_blank">dank@kegel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">With gitpoller, it was easy to see; whenever the number of git<br>
sessions from the poller went over 0 or so, web gui performance was<br>
poor.<br>
And if it went over 10, well, you could kiss the gui goodbye for<br>
several minutes.<br>
<br>
One countermeasure was to randomize the polling intervals, a la<br>
<br>
interval=6 # minutes<br>
self['change_source'].append(<br>
# Fuzz the interval to avoid slamming the git server<br>
and hitting the MaxStartups or MaxSessions limits<br>
# If you hit them, twistd.log will have lots of<br>
"ssh_exchange_identification: Connection closed by remote host" errors<br>
# See <a href="http://trac.buildbot.net/ticket/2480" rel="noreferrer" target="_blank">http://trac.buildbot.net/ticket/2480</a><br>
changes.GitPoller(repourl, branches=branchnames,<br>
workdir='gitpoller-workdir-'+name, pollinterval=interval*60 +<br>
random.uniform(-10, 10)))<br>
<br>
That made life just barely bearable, at least until number of projects<br>
polled was under 50 or so.<br>
What really helped was not using pollers anymore, and switching to<br>
gitlab's webhooks.<br>
We're at 190 now, of which 57 are still using gitpoller, and it's<br>
almost ok. (I really have<br>
to move the last 57 onto gitlab. Or, well, since they're not<br>
critical, increase the polling<br>
interval...)<br>
<div><div><br>
On Tue, Aug 2, 2016 at 9:13 AM, Pierre Tardy <<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>> wrote:<br>
> Hi,<br>
><br>
> Pollers are usually indeed not scaling as they, hmm, poll.<br>
> What you are describing here is hints that the twisted reactor thread is<br>
> always busy, which should not happen if you only start 10 builds.<br>
> You might have some custom steps which are doing something heavily cpu bound<br>
> in the main thread.<br>
> What I usually do is to use statprof:<br>
> <a href="https://pypi.python.org/pypi/statprof/" rel="noreferrer" target="_blank">https://pypi.python.org/pypi/statprof/</a><br>
><br>
> in order to know what the cpu is doing.<br>
> You could create a builder which you can trig whenever you need, and which<br>
> would start the profiling, wait a few minutes, and then save profiling to a<br>
> file.<br>
><br>
><br>
><br>
> Le mar. 2 août 2016 à 17:53, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>><br>
> a écrit :<br>
>><br>
>> Hey Dan,<br>
>><br>
>> I am using a p4 poller. Maybe it's suffering from the same problems?<br>
>><br>
>> On Tue, Aug 2, 2016 at 5:45 PM, Francesco Di Mizio<br>
>> <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>> wrote:<br>
>>><br>
>>> I'd like to provide a bit more context.Right after restarting the master<br>
>>> and kicking off 10 builds CPU was at 110-120%. This made the UI unusable and<br>
>>> basically all the services were stuck, including the REST API.<br>
>>> After 3-4 minutes like this and WITH all the 10 builds still running the<br>
>>> CPU usage went down to 5%, stayed there for 5 minutes and all was smooth and<br>
>>> quick again. From then on it keps oscillating, I've seen spikes of 240% :(<br>
>>><br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> On Tue, Aug 2, 2016 at 4:12 PM, Francesco Di Mizio<br>
>>> <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>> wrote:<br>
>>>><br>
>>>> Sometimes it goes up to 140%. I was not able to relate this with a<br>
>>>> particular builds condition - seems like it can happen any time and is not<br>
>>>> related to how many builds are going on.<br>
>>>><br>
>>>> I usually realize the server got into this state because the web UI gets<br>
>>>> stuck. As soon as the CPU% goes back to normal values (2-3% most times) the<br>
>>>> web finishes loading just instantly.<br>
>>>><br>
>>>> Any pointers as to what might be causing this? Only reason I can think<br>
>>>> of is too many people trying to access the web UI simultaniously - may I be<br>
>>>> right?<br>
>>>><br>
>>><br>
>><br>
>> _______________________________________________<br>
>> users mailing list<br>
>> <a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a><br>
>> <a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br>
><br>
><br>
> _______________________________________________<br>
> users mailing list<br>
> <a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a><br>
> <a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br>
</div></div></blockquote></div><br></div>
</blockquote></div>