<div dir="ltr">Pierre,<div><br></div><div>I will have IT move the server to a real physical box. Will be using a speedy SSD so if that's really the problem it should be evident soon.</div><div>Will of course update here with my findings.</div><div><br></div><div>Thanks!</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 16, 2016 at 12:59 PM, Pierre Tardy <span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Francesco,<div>I am continuing to do my scalability benchmarking with 0.9.2, and one issue I am finding is that buildbot consumes lots of memory in order to compress the logs.</div><div>It actually loads the whole log in memory in order to compress it at once</div><div>This simplifies the algorithm, but can be quite memory hungry. On my current test, I am doing 300 builds in parallel with 10k lines of log, and it eats 4GB+ of memory.</div><div>This explains why Greg had issue with 32bit python, and could explain your performance issue, if your master starts swapping while doing the compressions.</div><div><br></div><div>This is probably something that I want to eventually fix.</div><span class="HOEnZb"><font color="#888888"><div>Pierre</div><div><br></div></font></span></div><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote"><div dir="ltr">Le mer. 14 déc. 2016 à 19:12, Greg MacDonald <<a href="mailto:gmacdonald@trionworlds.com" target="_blank">gmacdonald@trionworlds.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d" class="m_-5088934517213390833gmail_msg">Might not be related but I had some issues with logs and UI lag when I was using 32 bit python for the server. Went away when I upgraded to 64 bit. - Greg<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></span></p>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d" class="m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></span></p>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><b class="m_-5088934517213390833gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="m_-5088934517213390833gmail_msg">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="m_-5088934517213390833gmail_msg"> users [mailto:<a href="mailto:users-bounces@buildbot.net" class="m_-5088934517213390833gmail_msg" target="_blank">users-bounces@<wbr>buildbot.net</a>]
<b class="m_-5088934517213390833gmail_msg">On Behalf Of </b>Pierre Tardy<br class="m_-5088934517213390833gmail_msg">
<b class="m_-5088934517213390833gmail_msg">Sent:</b> Wednesday, December 14, 2016 10:48 AM<br class="m_-5088934517213390833gmail_msg">
<b class="m_-5088934517213390833gmail_msg">To:</b> Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">francescodimizio@gmail.com</a>><br class="m_-5088934517213390833gmail_msg">
<b class="m_-5088934517213390833gmail_msg">Cc:</b> Buildbot <<a href="mailto:users@buildbot.net" class="m_-5088934517213390833gmail_msg" target="_blank">users@buildbot.net</a>><br class="m_-5088934517213390833gmail_msg">
<b class="m_-5088934517213390833gmail_msg">Subject:</b> Re: [<a href="mailto:users@bb.net" class="m_-5088934517213390833gmail_msg" target="_blank">users@bb.net</a>] buildbot CPU usage<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></span></p>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg"></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Hi Francesco,<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I have tried to reproduce your issue by doing some stress tests on the buildbot master.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I have been able to run thousands of builds throwing 30k lines of logs each without significant lag in the UI.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">it takes ~2min to compress the whole log with sqlite, but I can always get smooth access to the UI.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I also tried with pg, and it was much faster.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I am setting up a cleaner test environment, and I will be publishing some benchmarks.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">This week we got this bug about performance issue: <a href="http://trac.buildbot.net/ticket/3654#comment:4" class="m_-5088934517213390833gmail_msg" target="_blank">http://trac.buildbot.<wbr>net/ticket/3654#comment:4</a><u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">This eventually resolved to be a hardware problem on the machine. Could this be something like this?<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Pierre<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Le mar. 13 déc. 2016 à 01:26, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">francescodimizio@gmail.com</a>> a écrit :<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Pierre i am not using SQLite but pg hosted in an other container on the same box. I'd say the issue isn't which dB engine i am running as it also happens with MySQL.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Thanks,<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"> Francesco<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
</blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">On Dec 13, 2016 12:40 AM, "Pierre Tardy" <<a href="mailto:tardyp@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">tardyp@gmail.com</a>> wrote:<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Hi Francesco,<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I would say it is normal that the CPU goes to 100% while the log are being compressed. As soon as anything else is not blocked, for me this is fine.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I can see you are using sqlite. The thing is with sqlite the number of parallel access to the db is restricted to one by default in order to protect against weird parallel behaviour we got with sqlite.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I think this explains your behaviour.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I would recommend as an easy fix to switch to pg or mysql. I'll try to think of some workarounds like compressing the log only when there is a low load average. This is already something that we though about with my teammates. Log compression
is important, but not urgent, so this is fine to delay until there is not so much to do. <u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Pierre<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Le lun. 12 déc. 2016 à 12:40, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">francescodimizio@gmail.com</a>> a écrit :<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Hey Pierre,<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">first of all thanks for taking a look. I never ever have more than 9-10 builds going on at the same time. Like I said many times the system can happily be sitting on a nice 10-15% CPU usage when even 8 builds are running, but the CPU spikes
can happen at any time with any number of builds going on. Things seem to go worse when the system runs for a few days with no reboot. Just now I had 7 builds running and the cpu keeps oscillating between 10% and 120%, now it's contantly over 100%. I wonder
if we can find a way for you to access the system so you can take a look yourself (of course if you are fine with it)?<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I will def be looking at multimaster in the future, just have no time atm.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Thanks again,<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"> Francesco<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
</blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">On Mon, Dec 12, 2016 at 11:18 AM, Pierre Tardy <span class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119gmailmsg m_-5088934517213390833gmail_msg">
<<a href="mailto:tardyp@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">tardyp@gmail.com</a>></span> wrote:<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Hi Francesco,<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I finally got the time to look at your profile.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I dont see anything other than normal load. Buildbot is hardworking at compressing the log in the db, managing the build and steps, etc.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">How many build in parallel do you have for your master? Maybe its time to start thinking about multimaster.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Regards,<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Pierre<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
</blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Le lun. 28 nov. 2016 à 11:30, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">francescodimizio@gmail.com</a>> a écrit :<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Reviving this old thread. According to things that were said on this thread, I reduced as much as I could all the logs produced by all steps. Bigger has now 300 lines.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Despite that I am still seeing cpu spiked. I managed to grab a hopefully useful json from the profiler which you'll find attached. Toards the end you can clearly see the load goes through the roof.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">If anybody could take a look I'd really appreciate!<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Thanks,<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Francesco<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
</blockquote></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">On Fri, Aug 26, 2016 at 6:06 PM, Neil Gilmore <span class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649gmailmsg m_-5088934517213390833gmail_msg">
<<a href="mailto:ngilmore@grammatech.com" class="m_-5088934517213390833gmail_msg" target="_blank">ngilmore@grammatech.com</a>></span> wrote:<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Hi Pierre,<br class="m_-5088934517213390833gmail_msg">
<br class="m_-5088934517213390833gmail_msg">
Well, my previous message was really saying that I think buildbot can handle logs a lot bigger then 12K lines.<br class="m_-5088934517213390833gmail_msg">
<br class="m_-5088934517213390833gmail_msg">
It seems unlikely that it's stuck on log compression. The step says it's been running for 9.5 hours, and previous runs of the entire build have durations in the 1-3 hour range. The particular step is a ShellCommand that actually does the build. The place where
the log stops is during an svn up operation.<br class="m_-5088934517213390833gmail_msg">
<br class="m_-5088934517213390833gmail_msg">
I don't find any exceptions in the log for this, though that's not definitive (current number of log files is 122, and sometimes finding things is tricky). I do see the build cancellation in the log, and it also shows up in the UI.<br class="m_-5088934517213390833gmail_msg">
<br class="m_-5088934517213390833gmail_msg">
I'm betting on a lost deferred. So far, I've never seen an exception relating to that.<br class="m_-5088934517213390833gmail_msg">
<br class="m_-5088934517213390833gmail_msg">
On the bright side, this seems to be the only worker whose builders are stuck (some of the rest are waiting for locks, which this build has). Usually, there's more.<span style="color:#888888" class="m_-5088934517213390833gmail_msg"><br class="m_-5088934517213390833gmail_msg">
<br class="m_-5088934517213390833gmail_msg">
<span class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649m1048118493396790736hoenzb m_-5088934517213390833gmail_msg">Neil Gilmore</span><br class="m_-5088934517213390833gmail_msg">
<span class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649m1048118493396790736hoenzb m_-5088934517213390833gmail_msg"><a href="http://grammatech.com" class="m_-5088934517213390833gmail_msg" target="_blank">grammatech.com</a></span></span><u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div></blockquote></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg" style="margin-bottom:12.0pt"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">On 8/26/2016 10:35 AM, Pierre Tardy wrote:<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div></div></div></blockquote></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Hum, <u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">It might not be stuck, actually, but just spending very long time to compress the log.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">In theory, the log compression is not waited for, though.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Note that if you stop a build that is waiting for a lost deferred, this will have no effect as you describe.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">You got absolutly no exception in twisted.log?<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Pierre<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</blockquote></div></div></div></blockquote></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Le ven. 26 août 2016 à 17:06, Neil Gilmore <<a href="mailto:ngilmore@grammatech.com" class="m_-5088934517213390833gmail_msg" target="_blank">ngilmore@grammatech.com</a>> a écrit :<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div></blockquote></div></div></div></blockquote></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I'm currently looking at a step with 3 logs:<br class="m_-5088934517213390833gmail_msg">
81531 lines<br class="m_-5088934517213390833gmail_msg">
489285 lines<br class="m_-5088934517213390833gmail_msg">
489311 lines<br class="m_-5088934517213390833gmail_msg">
<br class="m_-5088934517213390833gmail_msg">
An earlier successful run would have that first log at 244080 lines.<br class="m_-5088934517213390833gmail_msg">
<br class="m_-5088934517213390833gmail_msg">
This particular build is stuck, though. :( (which is why I'm looking at it.) (A bit off-topic, but I tried stopping it. The last step is marked cancelled, but that's the only effect.)<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><br class="m_-5088934517213390833gmail_msg">
<br class="m_-5088934517213390833gmail_msg">
Neil Gilmore<br class="m_-5088934517213390833gmail_msg">
<a href="http://grammatech.com" class="m_-5088934517213390833gmail_msg" target="_blank">grammatech.com</a><u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg" style="margin-bottom:12.0pt"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">On 8/26/2016 6:25 AM, Pierre Tardy wrote:<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
<div class="m_-5088934517213390833gmail_msg">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg">The ram looks like sufficient, it might be a good test to try and increase the number of cpu for that VM.
<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">In your trace, I can see the use of up to 7 threads at the same time, so you might gain by going 8 CPUs<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Also, make sure that your VM host is not overbooked. In my experience of using VMware VMs provided by IT, overbooking has been a source of inexplicable performance issues.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">12k lines is a lot, but buildbot shall support this kind of load without issue.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Le ven. 26 août 2016 à 12:05, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">francescodimizio@gmail.com</a>> a écrit :<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">It's a vmware virtual machine with 4 GIGs RAM and 4 CPUs@3Ghz. It runs, among other marginal things, 2 docker containers - one for the buildbot and one for the postgres db.
<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">The most beefy logs have around 12K lines. Is it too much?<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Also some other logs are read from the worker's filesystem and added as additional logs.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">On Fri, Aug 26, 2016 at 11:51 AM, Pierre Tardy <span class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649gmailmsg m_-5088934517213390833gmail_msg">
<<a href="mailto:tardyp@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">tardyp@gmail.com</a>></span> wrote:<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Cool! <u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I can indeed see 3 spikes.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Looks related to logs and logs compression.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">What is the HW spec of your master machine?<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">How much log does your build generate?<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><span style="color:#888888" class="m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></span></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><span style="color:#888888" class="m_-5088934517213390833gmail_msg">Pierre<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></span></p>
</div>
</div>
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Le ven. 26 août 2016 à 11:42, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">francescodimizio@gmail.com</a>> a écrit :<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Pierre, <u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I enabled it, waited 1 min and saw the spike, then stopped after a few secs. Attached the json.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Awesome tool btw, work wonders!<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">On Thu, Aug 25, 2016 at 1:14 PM, Pierre Tardy <span class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649gmailmsg m_-5088934517213390833gmail_msg">
<<a href="mailto:tardyp@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">tardyp@gmail.com</a>></span> wrote:<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">You can try to hit the button in the morning with a two hours gather period, and hope that you see the spike during that period..
<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Le jeu. 25 août 2016 à 12:17, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">francescodimizio@gmail.com</a>> a écrit :<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Thanks a lot! Pierre I will def will give it a shot. I am not sure I'll be able to smash that 'start recording' button as the UI isusuallystuck when the CPU spikes. Updates to come!<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">On Thu, Aug 25, 2016 at 10:45 AM, Pierre Tardy <span class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649gmailmsg m_-5088934517213390833gmail_msg">
<<a href="mailto:tardyp@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">tardyp@gmail.com</a>></span> wrote:<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Hi Francesco, <u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I spent some time in order to implement a profiler plugin for buildbot<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">You can give it a look, and send your profile.json file if you need more analysis from me.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><a href="https://github.com/tardyp/buildbot_profiler" class="m_-5088934517213390833gmail_msg" target="_blank">https://github.com/tardyp/<wbr>buildbot_profiler</a><u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Regards,<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Pierre<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Le mer. 24 août 2016 à 22:43, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">francescodimizio@gmail.com</a>> a écrit :<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I've tried and it's not an easy task because of my Win into Vagrant into Docker setup.
<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I'll try again soon when I get a Linux box!<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">On Fri, Aug 19, 2016 at 5:54 PM, Vasily <span class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649gmailmsg m_-5088934517213390833gmail_msg">
<<a href="mailto:just.one.man@yandex.ru" class="m_-5088934517213390833gmail_msg" target="_blank">just.one.man@yandex.ru</a>></span> wrote:<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<p class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649gmailmsg1 m_-5088934517213390833gmail_msg">
Umm, no. VTune has Python support starting 2017 Beta, and, well, it was my team (at Intel) work actually :-)<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<p class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649gmailmsg1 m_-5088934517213390833gmail_msg">
P.S. I'm from Intel, too. <u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<p class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649gmailmsg1 m_-5088934517213390833gmail_msg">
Thanks, <br class="m_-5088934517213390833gmail_msg">
Vasily<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">19 авг. 2016 г. 18:17 пользователь "Francesco Di Mizio" <<a href="mailto:francescodimizio@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">francescodimizio@gmail.com</a>> написал:
<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<p class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649gmailmsg1 m_-5088934517213390833gmail_msg">
I had thought you were making fun of Intel somehow ;)<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">On Aug 19, 2016 5:07 PM, "Pierre Tardy" <<a href="mailto:tardyp@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">tardyp@gmail.com</a>> wrote:<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">ahah <u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I though this was a taunt on me being employed by Intel.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">I actually had mitigated experience with vtune few years ago, and didn't know they had python support until then.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Being an opensource guy, I usually neglegate to look at proprietary stuff.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><span style="color:#888888" class="m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></span></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><span style="color:#888888" class="m_-5088934517213390833gmail_msg">Pierre<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></span></p>
</div>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Le ven. 19 août 2016 à 12:18, Vasily <<a href="mailto:just.one.man@yandex.ru" class="m_-5088934517213390833gmail_msg" target="_blank">just.one.man@yandex.ru</a>> a écrit :<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<p class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649gmailmsg1 m_-5088934517213390833gmail_msg">
I'm again suggesting to look into Python profiling capabilities of Intel® VTune™ Amplifier. It could run statistical profiling for a long time and display CPU usage over time, so the developer can look at specific time range where CPU usage was too high and
see which functions were executed. <u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<p class="m_-5088934517213390833m_-8400205148090100180m-6213475157165445228m1616186179029383714m2129104286731852119m-3967462307026311269m-8715384078573251649gmailmsg1 m_-5088934517213390833gmail_msg">
Thanks, <br class="m_-5088934517213390833gmail_msg">
Vasily<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">19 авг. 2016 г. 11:57 пользователь "Pierre Tardy" <<a href="mailto:tardyp@gmail.com" class="m_-5088934517213390833gmail_msg" target="_blank">tardyp@gmail.com</a>> написал:<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Hi Francesco, <u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Your described setup looks sane to me.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">The problems we are trying to catch are cpu spikes, as far as I understand, which does not happen for very long, but are very annoying for users, as it is blocking the reactor.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">This problem is not easy to see in the profile you sent, as this profile is over long time, so we see the average of each method during the day and not the spikes.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">What would really be needed is a on-demand profiler which would detect cpu spikes and only log the stack traces during those times.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Here is a nice blog pst explaining why statistic profiling is cool and easy to implement in python.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><a href="https://nylas.com/blog/performance" class="m_-5088934517213390833gmail_msg" target="_blank">https://nylas.com/blog/<wbr>performance</a><u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">For 0.9.1 I want to concentrate on scalability, and write a debugging ui plugin based on those ideas (and probably code)<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">That would be great if your team can help on that matter.<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Regards,<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
<div class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">Pierre<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal m_-5088934517213390833gmail_msg" style="margin-bottom:12.0pt"><u class="m_-5088934517213390833gmail_msg"></u> <u class="m_-5088934517213390833gmail_msg"></u></p>
</blockquote>
</div>
</blockquote></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="m_-5088934517213390833gmail_msg">
<pre class="m_-5088934517213390833gmail_msg">______________________________<wbr>_________________<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></pre>
<pre class="m_-5088934517213390833gmail_msg">users mailing list<u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></pre>
<pre class="m_-5088934517213390833gmail_msg"><a href="mailto:users@buildbot.net" class="m_-5088934517213390833gmail_msg" target="_blank">users@buildbot.net</a><u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></pre>
<pre class="m_-5088934517213390833gmail_msg"><a href="https://lists.buildbot.net/mailman/listinfo/users" class="m_-5088934517213390833gmail_msg" target="_blank">https://lists.buildbot.net/<wbr>mailman/listinfo/users</a><u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></pre>
</blockquote>
</div></blockquote></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg">______________________________<wbr>_________________<br class="m_-5088934517213390833gmail_msg">
users mailing list<br class="m_-5088934517213390833gmail_msg">
<a href="mailto:users@buildbot.net" class="m_-5088934517213390833gmail_msg" target="_blank">users@buildbot.net</a><br class="m_-5088934517213390833gmail_msg">
<a href="https://lists.buildbot.net/mailman/listinfo/users" class="m_-5088934517213390833gmail_msg" target="_blank">https://lists.buildbot.net/<wbr>mailman/listinfo/users</a><u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</blockquote></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833m_-8400205148090100180WordSection1 m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><div class="m_-5088934517213390833gmail_msg"><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="m_-5088934517213390833gmail_msg">
<p class="MsoNormal m_-5088934517213390833gmail_msg"><br class="m_-5088934517213390833gmail_msg">
______________________________<wbr>_________________<br class="m_-5088934517213390833gmail_msg">
users mailing list<br class="m_-5088934517213390833gmail_msg">
<a href="mailto:users@buildbot.net" class="m_-5088934517213390833gmail_msg" target="_blank">users@buildbot.net</a><br class="m_-5088934517213390833gmail_msg">
<a href="https://lists.buildbot.net/mailman/listinfo/users" class="m_-5088934517213390833gmail_msg" target="_blank">https://lists.buildbot.net/<wbr>mailman/listinfo/users</a><u class="m_-5088934517213390833gmail_msg"></u><u class="m_-5088934517213390833gmail_msg"></u></p>
</blockquote></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div></blockquote></div>
</div></div></blockquote></div><br></div>