<div dir="ltr"><div dir="ltr">Please avoid sending big profiles to the mailing list. you can send it to me, or put a link to hosting server.<div><br></div><div>In your profile, I do see a lot of banana an readFD. banana is the serialisation protocol of twisted perspective broker.</div><div>This means that a lot of cpu is spent serializing communication to worker. My guts feeling is that there is an abuse of Download or Upload steps.</div><div>You shouldn't use this for too big files, and rather goes back to Artifactory or ssh protocols for large file transfers.</div><div><br></div><div>Regards<br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Pierre</div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le jeu. 14 janv. 2021 à 10:11, <<a href="mailto:users-bounces@buildbot.net">users-bounces@buildbot.net</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Reviving this old thread. According to things that were said on this thread, I reduced as much as I could all the logs produced by all steps. Bigger has now 300 lines.<div>Despite that I am still seeing cpu spiked. I managed to grab a hopefully useful json from the profiler which you'll find attached. Toards the end you can clearly see the load goes through the roof.</div><div><br></div><div>If anybody could take a look I'd really appreciate!</div><div><br></div><div>Thanks,</div><div>Francesco</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 26, 2016 at 6:06 PM, Neil Gilmore <span dir="ltr"><<a href="mailto:ngilmore@grammatech.com" target="_blank">ngilmore@grammatech.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
Hi Pierre,<br>
<br>
Well, my previous message was really saying that I think buildbot
can handle logs a lot bigger then 12K lines.<br>
<br>
It seems unlikely that it's stuck on log compression. The step says
it's been running for 9.5 hours, and previous runs of the entire
build have durations in the 1-3 hour range. The particular step is a
ShellCommand that actually does the build. The place where the log
stops is during an svn up operation.<br>
<br>
I don't find any exceptions in the log for this, though that's not
definitive (current number of log files is 122, and sometimes
finding things is tricky). I do see the build cancellation in the
log, and it also shows up in the UI.<br>
<br>
I'm betting on a lost deferred. So far, I've never seen an exception
relating to that.<br>
<br>
On the bright side, this seems to be the only worker whose builders
are stuck (some of the rest are waiting for locks, which this build
has). Usually, there's more.<span><font color="#888888"><br>
<br>
Neil Gilmore<br>
<a href="http://grammatech.com" target="_blank">grammatech.com</a></font></span><div><div><br>
<br>
<div>On 8/26/2016 10:35 AM, Pierre Tardy
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hum,
<div>It might not be stuck, actually, but just spending very
long time to compress the log.</div>
<div><br>
</div>
<div>In theory, the log compression is not waited for, though.</div>
<div><br>
</div>
<div>Note that if you stop a build that is waiting for a lost
deferred, this will have no effect as you describe.</div>
<div><br>
</div>
<div>You got absolutly no exception in twisted.log?</div>
<div><br>
</div>
<div>Pierre</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Le ven. 26 août 2016 à 17:06, Neil Gilmore <<a href="mailto:ngilmore@grammatech.com" target="_blank">ngilmore@grammatech.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> I'm currently looking
at a step with 3 logs:<br>
81531 lines<br>
489285 lines<br>
489311 lines<br>
<br>
An earlier successful run would have that first log at
244080 lines.<br>
<br>
This particular build is stuck, though. :( (which is why I'm
looking at it.) (A bit off-topic, but I tried stopping it.
The last step is marked cancelled, but that's the only
effect.)</div>
<div bgcolor="#FFFFFF"><br>
<br>
Neil Gilmore<br>
<a href="http://grammatech.com" target="_blank">grammatech.com</a></div>
<div bgcolor="#FFFFFF"><br>
<br>
<div>On 8/26/2016 6:25 AM, Pierre Tardy wrote:<br>
</div>
</div>
<div bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
The ram looks like sufficient, it might be a good test
to try and increase the number of cpu for that VM.
<div>In your trace, I can see the use of up to 7 threads
at the same time, so you might gain by going 8 CPUs</div>
<div><br>
</div>
<div>Also, make sure that your VM host is not
overbooked. In my experience of using VMware VMs
provided by IT, overbooking has been a source of
inexplicable performance issues.</div>
<div><br>
</div>
<div>12k lines is a lot, but buildbot shall support this
kind of load without issue.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Le ven. 26 août 2016 à 12:05, Francesco
Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>> a
écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">It's a vmware virtual machine with 4
GIGs RAM and 4 CPUs@3Ghz. It runs, among other
marginal things, 2 docker containers - one for the
buildbot and one for the postgres db.
<div><br>
</div>
<div>The most beefy logs have around 12K lines. Is
it too much?</div>
<div>Also some other logs are read from the worker's
filesystem and added as additional logs.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Aug 26, 2016 at
11:51 AM, Pierre Tardy <span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Cool!
<div>I can indeed see 3 spikes.</div>
<div><br>
</div>
<div>Looks related to logs and logs
compression.</div>
<div><br>
</div>
<div>What is the HW spec of your master
machine?</div>
<div>How much log does your build generate?</div>
<span><font color="#888888">
<div><br>
</div>
<div>Pierre</div>
</font></span></div>
<div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">Le ven. 26 août 2016
à 11:42, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Pierre,
<div><br>
</div>
<div>I enabled it, waited 1 min and
saw the spike, then stopped after a
few secs. Attached the json.</div>
<div>Awesome tool btw, work wonders!</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Aug
25, 2016 at 1:14 PM, Pierre Tardy <span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">You can try to hit
the button in the morning with a
two hours gather period, and
hope that you see the spike
during that period..
<div><br>
</div>
</div>
<div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">Le jeu. 25
août 2016 à 12:17,
Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Thanks a
lot! Pierre I will def
will give it a shot. I
am not sure I'll be able
to smash that 'start
recording' button as the
UI isusuallystuck when
the CPU spikes. Updates
to come!</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Aug 25, 2016 at
10:45 AM, Pierre Tardy
<span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi
Francesco,
<div><br>
</div>
<div>I spent some
time in order to
implement a
profiler plugin
for buildbot</div>
<div><br>
</div>
<div>You can give
it a look, and
send your
profile.json
file if you need
more analysis
from me.</div>
<div>
<div><a href="https://github.com/tardyp/buildbot_profiler" target="_blank">https://github.com/tardyp/buildbot_profiler</a><br>
</div>
<div><br>
</div>
<div>Regards,</div>
<div>Pierre</div>
<br>
</div>
</div>
<div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">Le mer.
24 août 2016
à 22:43,
Francesco Di
Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">I've
tried and it's
not an easy
task because
of my Win into
Vagrant into
Docker setup.
<div>I'll try
again soon
when I get a
Linux box!</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Fri, Aug 19,
2016 at 5:54
PM, Vasily <span dir="ltr"><<a href="mailto:just.one.man@yandex.ru" target="_blank">just.one.man@yandex.ru</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<p dir="ltr">Umm,
no. VTune has
Python support
starting 2017
Beta, and,
well, it was
my team (at
Intel) work
actually :-)</p>
<p dir="ltr">P.S.
I'm from
Intel, too. </p>
<p dir="ltr">Thanks,
<br>
Vasily</p>
<div class="gmail_quote">19
авг. 2016 г.
18:17
пользователь
"Francesco Di
Mizio" <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>>
написал:
<div>
<div><br type="attribution">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<p dir="ltr">I
had thought
you were
making fun of
Intel somehow
;)</p>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Aug 19, 2016
5:07 PM,
"Pierre Tardy"
<<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>>
wrote:<br type="attribution">
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">ahah
<div><br>
</div>
<div>I though
this was a
taunt on me
being employed
by Intel.</div>
<div>I
actually had
mitigated
experience
with vtune few
years ago, and
didn't know
they had
python support
until then.</div>
<div>Being an
opensource
guy, I usually
neglegate to
look at
proprietary
stuff.</div>
<font color="#888888">
<div><br>
</div>
<div>Pierre</div>
</font></div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">Le ven.
19 août 2016
à 12:18,
Vasily <<a href="mailto:just.one.man@yandex.ru" target="_blank">just.one.man@yandex.ru</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<p dir="ltr">I'm
again
suggesting to
look into
Python
profiling
capabilities
of Intel®
VTune™
Amplifier. It
could run
statistical
profiling for
a long time
and display
CPU usage over
time, so the
developer can
look at
specific time
range where
CPU usage was
too high and
see which
functions were
executed. </p>
<p dir="ltr">Thanks,
<br>
Vasily</p>
<div class="gmail_quote">19
авг. 2016 г.
11:57
пользователь
"Pierre Tardy"
<<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>>
написал:</div>
<div class="gmail_quote"><br type="attribution">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi
Francesco,
<div><br>
</div>
<div>Your
described
setup looks
sane to me.</div>
<div><br>
</div>
<div>The
problems we
are trying to
catch are cpu
spikes, as far
as I
understand,
which does not
happen for
very long, but
are very
annoying for
users, as it
is blocking
the reactor.</div>
<div><br>
</div>
<div>This
problem is not
easy to see in
the profile
you sent, as
this profile
is over long
time, so we
see the
average of
each method
during the day
and not the
spikes.</div>
<div><br>
</div>
<div>What
would really
be needed is a
on-demand
profiler which
would detect
cpu spikes and
only log the
stack traces
during those
times.</div>
<div><br>
</div>
<div>Here is a
nice blog pst
explaining why
statistic
profiling is
cool and easy
to implement
in python.</div>
<div><a href="https://nylas.com/blog/performance" target="_blank">https://nylas.com/blog/performance</a><br>
</div>
<div><br>
</div>
<div>For 0.9.1
I want to
concentrate on
scalability,
and write a
debugging ui
plugin based
on those ideas
(and probably
code)</div>
<div><br>
</div>
<div>That
would be great
if your team
can help on
that matter.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Pierre</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<br>
</blockquote>
</div>
<div bgcolor="#FFFFFF">
<blockquote type="cite">
<pre>_______________________________________________
users mailing list
<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a>
<a href="https://lists.buildbot.net/mailman/listinfo/users" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a></pre>
</blockquote>
</div>
_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a><br>
<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a></blockquote>
</div>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a><br>
<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br></blockquote></div><br></div>
</blockquote></div></div>