<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
I'm currently looking at a step with 3 logs:<br>
81531 lines<br>
489285 lines<br>
489311 lines<br>
<br>
An earlier successful run would have that first log at 244080 lines.<br>
<br>
This particular build is stuck, though. :( (which is why I'm looking
at it.) (A bit off-topic, but I tried stopping it. The last step is
marked cancelled, but that's the only effect.)<br>
<br>
Neil Gilmore<br>
grammatech.com<br>
<br>
<div class="moz-cite-prefix">On 8/26/2016 6:25 AM, Pierre Tardy
wrote:<br>
</div>
<blockquote
cite="mid:CAJ+soVcLRiniVCj849WPvGZpE3A+QiPY_sUBxqPJjSdJv9+pLw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
The ram looks like sufficient, it might be a good test to try
and increase the number of cpu for that VM.
<div>In your trace, I can see the use of up to 7 threads at the
same time, so you might gain by going 8 CPUs</div>
<div><br>
</div>
<div>Also, make sure that your VM host is not overbooked. In my
experience of using VMware VMs provided by IT, overbooking has
been a source of inexplicable performance issues.</div>
<div><br>
</div>
<div>12k lines is a lot, but buildbot shall support this kind of
load without issue.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Le ven. 26 août 2016 à 12:05, Francesco Di Mizio
<<a moz-do-not-send="true"
href="mailto:francescodimizio@gmail.com">francescodimizio@gmail.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">It's a vmware virtual machine with 4 GIGs RAM
and 4 CPUs@3Ghz. It runs, among other marginal things, 2
docker containers - one for the buildbot and one for the
postgres db.
<div><br>
</div>
<div>The most beefy logs have around 12K lines. Is it too
much?</div>
<div>Also some other logs are read from the worker's
filesystem and added as additional logs.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Aug 26, 2016 at 11:51 AM,
Pierre Tardy <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:tardyp@gmail.com"
target="_blank">tardyp@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Cool!
<div>I can indeed see 3 spikes.</div>
<div><br>
</div>
<div>Looks related to logs and logs compression.</div>
<div><br>
</div>
<div>What is the HW spec of your master machine?</div>
<div>How much log does your build generate?</div>
<span><font color="#888888">
<div><br>
</div>
<div>Pierre</div>
</font></span></div>
<div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">Le ven. 26 août 2016 à 11:42,
Francesco Di Mizio <<a moz-do-not-send="true"
href="mailto:francescodimizio@gmail.com"
target="_blank">francescodimizio@gmail.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">Pierre,
<div><br>
</div>
<div>I enabled it, waited 1 min and saw the
spike, then stopped after a few secs.
Attached the json.</div>
<div>Awesome tool btw, work wonders!</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Aug 25, 2016
at 1:14 PM, Pierre Tardy <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:tardyp@gmail.com"
target="_blank">tardyp@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div dir="ltr">You can try to hit the
button in the morning with a two hours
gather period, and hope that you see the
spike during that period..
<div><br>
</div>
</div>
<div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">Le jeu. 25 août 2016
à 12:17, Francesco Di Mizio <<a
moz-do-not-send="true"
href="mailto:francescodimizio@gmail.com"
target="_blank">francescodimizio@gmail.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">Thanks a lot!
Pierre I will def will give it a
shot. I am not sure I'll be able
to smash that 'start recording'
button as the UI isusuallystuck
when the CPU spikes. Updates to
come!</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu,
Aug 25, 2016 at 10:45 AM,
Pierre Tardy <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:tardyp@gmail.com"
target="_blank">tardyp@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">Hi Francesco,
<div><br>
</div>
<div>I spent some time in
order to implement a
profiler plugin for
buildbot</div>
<div><br>
</div>
<div>You can give it a
look, and send your
profile.json file if you
need more analysis from
me.</div>
<div>
<div><a
moz-do-not-send="true"
href="https://github.com/tardyp/buildbot_profiler" target="_blank">https://github.com/tardyp/buildbot_profiler</a><br>
</div>
<div><br>
</div>
<div>Regards,</div>
<div>Pierre</div>
<br>
</div>
</div>
<div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">Le mer.
24 août 2016
à 22:43, Francesco
Di Mizio <<a
moz-do-not-send="true"
href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>>
a écrit :<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">I've
tried and it's not
an easy task
because of my Win
into Vagrant into
Docker setup.
<div>I'll try
again soon when
I get a Linux
box!</div>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Fri, Aug 19,
2016 at 5:54 PM,
Vasily <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:just.one.man@yandex.ru"
target="_blank">just.one.man@yandex.ru</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr">Umm,
no. VTune has
Python support
starting 2017
Beta, and,
well, it was
my team (at
Intel) work
actually :-)</p>
<p dir="ltr">P.S.
I'm from
Intel, too. </p>
<p dir="ltr">Thanks,
<br>
Vasily</p>
<div
class="gmail_quote">19
авг. 2016 г.
18:17
пользователь
"Francesco Di
Mizio" <<a
moz-do-not-send="true" href="mailto:francescodimizio@gmail.com"
target="_blank">francescodimizio@gmail.com</a>>
написал:
<div>
<div><br
type="attribution">
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr">I
had thought
you were
making fun of
Intel somehow
;)</p>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Aug 19, 2016
5:07 PM,
"Pierre Tardy"
<<a
moz-do-not-send="true"
href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>>
wrote:<br
type="attribution">
<blockquote
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">ahah
<div><br>
</div>
<div>I though
this was a
taunt on me
being employed
by Intel.</div>
<div>I
actually had
mitigated
experience
with vtune few
years ago, and
didn't know
they had
python support
until then.</div>
<div>Being an
opensource
guy, I usually
neglegate to
look at
proprietary
stuff.</div>
<font
color="#888888">
<div><br>
</div>
<div>Pierre</div>
</font></div>
<div><br>
<div
class="gmail_quote">
<div dir="ltr">Le ven.
19 août 2016
à 12:18,
Vasily <<a
moz-do-not-send="true" href="mailto:just.one.man@yandex.ru"
target="_blank">just.one.man@yandex.ru</a>>
a écrit :<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr">I'm
again
suggesting to
look into
Python
profiling
capabilities
of Intel®
VTune™
Amplifier. It
could run
statistical
profiling for
a long time
and display
CPU usage over
time, so the
developer can
look at
specific time
range where
CPU usage was
too high and
see which
functions were
executed. </p>
<p dir="ltr">Thanks,
<br>
Vasily</p>
<div
class="gmail_quote">19
авг. 2016 г.
11:57
пользователь
"Pierre Tardy"
<<a
moz-do-not-send="true"
href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>>
написал:</div>
<div
class="gmail_quote"><br
type="attribution">
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi
Francesco,
<div><br>
</div>
<div>Your
described
setup looks
sane to me.</div>
<div><br>
</div>
<div>The
problems we
are trying to
catch are cpu
spikes, as far
as I
understand,
which does not
happen for
very long, but
are very
annoying for
users, as it
is blocking
the reactor.</div>
<div><br>
</div>
<div>This
problem is not
easy to see in
the profile
you sent, as
this profile
is over long
time, so we
see the
average of
each method
during the day
and not the
spikes.</div>
<div><br>
</div>
<div>What
would really
be needed is a
on-demand
profiler which
would detect
cpu spikes and
only log the
stack traces
during those
times.</div>
<div><br>
</div>
<div>Here is a
nice blog pst
explaining why
statistic
profiling is
cool and easy
to implement
in python.</div>
<div><a
moz-do-not-send="true"
href="https://nylas.com/blog/performance" target="_blank">https://nylas.com/blog/performance</a><br>
</div>
<div><br>
</div>
<div>For 0.9.1
I want to
concentrate on
scalability,
and write a
debugging ui
plugin based
on those ideas
(and probably
code)</div>
<div><br>
</div>
<div>That
would be great
if your team
can help on
that matter.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Pierre</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:users@buildbot.net">users@buildbot.net</a>
<a class="moz-txt-link-freetext" href="https://lists.buildbot.net/mailman/listinfo/users">https://lists.buildbot.net/mailman/listinfo/users</a></pre>
</blockquote>
<br>
</body>
</html>