<div dir="ltr">Hi Francesco,<div>I would say it is normal that the CPU goes to 100% while the log are being compressed. As soon as anything else is not blocked, for me this is fine.</div><div>I can see you are using sqlite. The thing is with sqlite the number of parallel access to the db is restricted to one by default in order to protect against weird parallel behaviour we got with sqlite.</div><div>I think this explains your behaviour.</div><div>I would recommend as an easy fix to switch to pg or mysql. I'll try to think of some workarounds like compressing the log only when there is a low load average. This is already something that we though about with my teammates. Log compression is important, but not urgent, so this is fine to delay until there is not so much to do. </div><div><br></div><div>Pierre</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">Le lun. 12 déc. 2016 à 12:40, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com">francescodimizio@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Hey Pierre,<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">first of all thanks for taking a look. I never ever have more than 9-10 builds going on at the same time. Like I said many times the system can happily be sitting on a nice 10-15% CPU usage when even 8 builds are running, but the CPU spikes can happen at any time with any number of builds going on. Things seem to go worse when the system runs for a few days with no reboot. Just now I had 7 builds running and the cpu keeps oscillating between 10% and 120%, now it's contantly over 100%. I wonder if we can find a way for you to access the system so you can take a look yourself (of course if you are fine with it)?</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I will def be looking at multimaster in the future, just have no time atm.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Thanks again,</div><div class="gmail_msg"> Francesco</div></div><div class="gmail_extra gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg">On Mon, Dec 12, 2016 at 11:18 AM, Pierre Tardy <span dir="ltr" class="gmail_msg"><<a href="mailto:tardyp@gmail.com" class="gmail_msg" target="_blank">tardyp@gmail.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Hi Francesco,<div class="gmail_msg">I finally got the time to look at your profile.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I dont see anything other than normal load. Buildbot is hardworking at compressing the log in the db, managing the build and steps, etc.</div><div class="gmail_msg">How many build in parallel do you have for your master? Maybe its time to start thinking about multimaster.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Regards,</div><div class="gmail_msg">Pierre</div></div><div class="m_-3967462307026311269HOEnZb gmail_msg"><div class="m_-3967462307026311269h5 gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">Le lun. 28 nov. 2016 à 11:30, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="gmail_msg" target="_blank">francescodimizio@gmail.com</a>> a écrit :<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Reviving this old thread. According to things that were said on this thread, I reduced as much as I could all the logs produced by all steps. Bigger has now 300 lines.<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Despite that I am still seeing cpu spiked. I managed to grab a hopefully useful json from the profiler which you'll find attached. Toards the end you can clearly see the load goes through the roof.</div><div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"></div><div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">If anybody could take a look I'd really appreciate!</div><div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"></div><div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Thanks,</div><div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Francesco</div></div><div class="gmail_extra m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">On Fri, Aug 26, 2016 at 6:06 PM, Neil Gilmore <span dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><<a href="mailto:ngilmore@grammatech.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">ngilmore@grammatech.com</a>></span> wrote:<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
Hi Pierre,<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
Well, my previous message was really saying that I think buildbot
can handle logs a lot bigger then 12K lines.<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
It seems unlikely that it's stuck on log compression. The step says
it's been running for 9.5 hours, and previous runs of the entire
build have durations in the 1-3 hour range. The particular step is a
ShellCommand that actually does the build. The place where the log
stops is during an svn up operation.<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
I don't find any exceptions in the log for this, though that's not
definitive (current number of log files is 122, and sometimes
finding things is tricky). I do see the build cancellation in the
log, and it also shows up in the UI.<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
I'm betting on a lost deferred. So far, I've never seen an exception
relating to that.<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
On the bright side, this seems to be the only worker whose builders
are stuck (some of the rest are waiting for locks, which this build
has). Usually, there's more.<span class="m_-3967462307026311269m_-8715384078573251649m_1048118493396790736HOEnZb m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><font color="#888888" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
Neil Gilmore<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<a href="http://grammatech.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">grammatech.com</a></font></span><div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><div class="m_-3967462307026311269m_-8715384078573251649m_1048118493396790736h5 m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="m_-3967462307026311269m_-8715384078573251649m_1048118493396790736m_1300710874298465658moz-cite-prefix m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">On 8/26/2016 10:35 AM, Pierre Tardy
wrote:<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<blockquote type="cite" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Hum,
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">It might not be stuck, actually, but just spending very
long time to compress the log.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">In theory, the log compression is not waited for, though.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Note that if you stop a build that is waiting for a lost
deferred, this will have no effect as you describe.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">You got absolutly no exception in twisted.log?</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Pierre</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
</div>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Le ven. 26 août 2016 à 17:06, Neil Gilmore <<a href="mailto:ngilmore@grammatech.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">ngilmore@grammatech.com</a>>
a écrit :<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"> I'm currently looking
at a step with 3 logs:<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
81531 lines<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
489285 lines<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
489311 lines<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
An earlier successful run would have that first log at
244080 lines.<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
This particular build is stuck, though. :( (which is why I'm
looking at it.) (A bit off-topic, but I tried stopping it.
The last step is marked cancelled, but that's the only
effect.)</div>
<div bgcolor="#FFFFFF" text="#000000" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
Neil Gilmore<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<a href="http://grammatech.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">grammatech.com</a></div>
<div bgcolor="#FFFFFF" text="#000000" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">On 8/26/2016 6:25 AM, Pierre Tardy wrote:<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
</div>
<div bgcolor="#FFFFFF" text="#000000" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<blockquote type="cite" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
The ram looks like sufficient, it might be a good test
to try and increase the number of cpu for that VM.
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">In your trace, I can see the use of up to 7 threads
at the same time, so you might gain by going 8 CPUs</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Also, make sure that your VM host is not
overbooked. In my experience of using VMware VMs
provided by IT, overbooking has been a source of
inexplicable performance issues.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">12k lines is a lot, but buildbot shall support this
kind of load without issue.</div>
</div>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Le ven. 26 août 2016 à 12:05, Francesco
Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">francescodimizio@gmail.com</a>> a
écrit :<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">It's a vmware virtual machine with 4
GIGs RAM and 4 CPUs@3Ghz. It runs, among other
marginal things, 2 docker containers - one for the
buildbot and one for the postgres db.
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">The most beefy logs have around 12K lines. Is
it too much?</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Also some other logs are read from the worker's
filesystem and added as additional logs.</div>
</div>
<div class="gmail_extra m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">On Fri, Aug 26, 2016 at
11:51 AM, Pierre Tardy <span dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><<a href="mailto:tardyp@gmail.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">tardyp@gmail.com</a>></span>
wrote:<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Cool!
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">I can indeed see 3 spikes.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Looks related to logs and logs
compression.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">What is the HW spec of your master
machine?</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">How much log does your build generate?</div>
<span class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><font color="#888888" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Pierre</div>
</font></span></div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Le ven. 26 août 2016
à 11:42, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">francescodimizio@gmail.com</a>>
a écrit :<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Pierre,
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">I enabled it, waited 1 min and
saw the spike, then stopped after a
few secs. Attached the json.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Awesome tool btw, work wonders!</div>
</div>
<div class="gmail_extra m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">On Thu, Aug
25, 2016 at 1:14 PM, Pierre Tardy <span dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><<a href="mailto:tardyp@gmail.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">tardyp@gmail.com</a>></span>
wrote:<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">You can try to hit
the button in the morning with a
two hours gather period, and
hope that you see the spike
during that period..
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Le jeu. 25
août 2016 à 12:17,
Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">francescodimizio@gmail.com</a>>
a écrit :<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Thanks a
lot! Pierre I will def
will give it a shot. I
am not sure I'll be able
to smash that 'start
recording' button as the
UI isusuallystuck when
the CPU spikes. Updates
to come!</div>
<div class="gmail_extra m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">On
Thu, Aug 25, 2016 at
10:45 AM, Pierre Tardy
<span dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><<a href="mailto:tardyp@gmail.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">tardyp@gmail.com</a>></span>
wrote:<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Hi
Francesco,
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">I spent some
time in order to
implement a
profiler plugin
for buildbot</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">You can give
it a look, and
send your
profile.json
file if you need
more analysis
from me.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><a href="https://github.com/tardyp/buildbot_profiler" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">https://github.com/tardyp/buildbot_profiler</a><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Regards,</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Pierre</div>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Le mer.
24 août 2016
à 22:43,
Francesco Di
Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">francescodimizio@gmail.com</a>>
a écrit :<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">I've
tried and it's
not an easy
task because
of my Win into
Vagrant into
Docker setup.
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">I'll try
again soon
when I get a
Linux box!</div>
</div>
<div class="gmail_extra m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">On
Fri, Aug 19,
2016 at 5:54
PM, Vasily <span dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><<a href="mailto:just.one.man@yandex.ru" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">just.one.man@yandex.ru</a>></span>
wrote:<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Umm,
no. VTune has
Python support
starting 2017
Beta, and,
well, it was
my team (at
Intel) work
actually :-)</p>
<p dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">P.S.
I'm from
Intel, too. </p>
<p dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Thanks,
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
Vasily</p>
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">19
авг. 2016 г.
18:17
пользователь
"Francesco Di
Mizio" <<a href="mailto:francescodimizio@gmail.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">francescodimizio@gmail.com</a>>
написал:
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br type="attribution" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">I
had thought
you were
making fun of
Intel somehow
;)</p>
<div class="gmail_extra m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">On
Aug 19, 2016
5:07 PM,
"Pierre Tardy"
<<a href="mailto:tardyp@gmail.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">tardyp@gmail.com</a>>
wrote:<br type="attribution" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">ahah
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">I though
this was a
taunt on me
being employed
by Intel.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">I
actually had
mitigated
experience
with vtune few
years ago, and
didn't know
they had
python support
until then.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Being an
opensource
guy, I usually
neglegate to
look at
proprietary
stuff.</div>
<font color="#888888" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Pierre</div>
</font></div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Le ven.
19 août 2016
à 12:18,
Vasily <<a href="mailto:just.one.man@yandex.ru" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">just.one.man@yandex.ru</a>>
a écrit :<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">I'm
again
suggesting to
look into
Python
profiling
capabilities
of Intel®
VTune™
Amplifier. It
could run
statistical
profiling for
a long time
and display
CPU usage over
time, so the
developer can
look at
specific time
range where
CPU usage was
too high and
see which
functions were
executed. </p>
<p dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Thanks,
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
Vasily</p>
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">19
авг. 2016 г.
11:57
пользователь
"Pierre Tardy"
<<a href="mailto:tardyp@gmail.com" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">tardyp@gmail.com</a>>
написал:</div>
<div class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br type="attribution" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<blockquote class="gmail_quote m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Hi
Francesco,
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Your
described
setup looks
sane to me.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">The
problems we
are trying to
catch are cpu
spikes, as far
as I
understand,
which does not
happen for
very long, but
are very
annoying for
users, as it
is blocking
the reactor.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">This
problem is not
easy to see in
the profile
you sent, as
this profile
is over long
time, so we
see the
average of
each method
during the day
and not the
spikes.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">What
would really
be needed is a
on-demand
profiler which
would detect
cpu spikes and
only log the
stack traces
during those
times.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Here is a
nice blog pst
explaining why
statistic
profiling is
cool and easy
to implement
in python.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><a href="https://nylas.com/blog/performance" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">https://nylas.com/blog/performance</a><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">For 0.9.1
I want to
concentrate on
scalability,
and write a
debugging ui
plugin based
on those ideas
(and probably
code)</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">That
would be great
if your team
can help on
that matter.</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Regards,</div>
<div class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">Pierre</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div>
</blockquote>
</div>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<fieldset class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"></fieldset>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</blockquote>
</div>
<div bgcolor="#FFFFFF" text="#000000" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<blockquote type="cite" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<pre class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">_______________________________________________
users mailing list
<a href="mailto:users@buildbot.net" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">users@buildbot.net</a>
<a href="https://lists.buildbot.net/mailman/listinfo/users" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a></pre>
</blockquote>
</div>
_______________________________________________<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
users mailing list<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<a href="mailto:users@buildbot.net" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">users@buildbot.net</a><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a></blockquote>
</div>
</blockquote>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
</div></div></div>
<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">_______________________________________________<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
users mailing list<br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<a href="mailto:users@buildbot.net" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">users@buildbot.net</a><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg">
<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"></blockquote></div><br class="m_-3967462307026311269m_-8715384078573251649gmail_msg gmail_msg"></div>
</blockquote></div>
</div></div></blockquote></div><br class="gmail_msg"></div>
</blockquote></div>