<div dir="ltr">Hum,<div>It might not be stuck, actually, but just spending very long time to compress the log.</div><div><br></div><div>In theory, the log compression is not waited for, though.</div><div><br></div><div>Note that if you stop a build that is waiting for a lost deferred, this will have no effect as you describe.</div><div><br></div><div>You got absolutly no exception in twisted.log?</div><div><br></div><div>Pierre</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">Le ven. 26 août 2016 à 17:06, Neil Gilmore <<a href="mailto:ngilmore@grammatech.com">ngilmore@grammatech.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    I'm currently looking at a step with 3 logs:<br>
    81531 lines<br>
    489285 lines<br>
    489311 lines<br>
    <br>
    An earlier successful run would have that first log at 244080 lines.<br>
    <br>
    This particular build is stuck, though. :( (which is why I'm looking
    at it.) (A bit off-topic, but I tried stopping it. The last step is
    marked cancelled, but that's the only effect.)</div><div bgcolor="#FFFFFF" text="#000000"><br>
    <br>
    Neil Gilmore<br>
    <a href="http://grammatech.com" target="_blank">grammatech.com</a></div><div bgcolor="#FFFFFF" text="#000000"><br>
    <br>
    <div>On 8/26/2016 6:25 AM, Pierre Tardy
      wrote:<br>
    </div>
    </div><div bgcolor="#FFFFFF" text="#000000"><blockquote type="cite">
      <div dir="ltr">
        <div><br>
        </div>
        The ram looks like sufficient, it might be a good test to try
        and increase the number of cpu for that VM.
        <div>In your trace, I can see the use of up to 7 threads at the
          same time, so you might gain by going 8 CPUs</div>
        <div><br>
        </div>
        <div>Also, make sure that your VM host is not overbooked. In my
          experience of using VMware VMs provided by IT, overbooking has
          been a source of inexplicable performance issues.</div>
        <div><br>
        </div>
        <div>12k lines is a lot, but buildbot shall support this kind of
          load without issue.</div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr">Le ven. 26 août 2016 à 12:05, Francesco Di Mizio
          <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>>
          a écrit :<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
          <div dir="ltr">It's a vmware virtual machine with 4 GIGs RAM
            and 4 CPUs@3Ghz. It runs, among other marginal things, 2
            docker containers - one for the buildbot and one for the
            postgres db.
            <div><br>
            </div>
            <div>The most beefy logs have around 12K lines. Is it too
              much?</div>
            <div>Also some other logs are read from the worker's
              filesystem and added as additional logs.</div>
          </div>
          <div class="gmail_extra"><br>
            <div class="gmail_quote">On Fri, Aug 26, 2016 at 11:51 AM,
              Pierre Tardy <span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span> wrote:<br>
              <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                <div dir="ltr">Cool!
                  <div>I can indeed see 3 spikes.</div>
                  <div><br>
                  </div>
                  <div>Looks related to logs and logs compression.</div>
                  <div><br>
                  </div>
                  <div>What is the HW spec of your master machine?</div>
                  <div>How much log does your build generate?</div>
                  <span><font color="#888888">
                      <div><br>
                      </div>
                      <div>Pierre</div>
                    </font></span></div>
                <div>
                  <div><br>
                    <div class="gmail_quote">
                      <div dir="ltr">Le ven. 26 août 2016 à 11:42,
                        Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>>
                        a écrit :<br>
                      </div>
                      <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                        <div dir="ltr">Pierre,
                          <div><br>
                          </div>
                          <div>I enabled it, waited 1 min and saw the
                            spike, then stopped after a few secs.
                            Attached the json.</div>
                          <div>Awesome tool btw, work wonders!</div>
                        </div>
                        <div class="gmail_extra"><br>
                          <div class="gmail_quote">On Thu, Aug 25, 2016
                            at 1:14 PM, Pierre Tardy <span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span>
                            wrote:<br>
                            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                              <div dir="ltr">You can try to hit the
                                button in the morning with a two hours
                                gather period, and hope that you see the
                                spike during that period..
                                <div><br>
                                </div>
                              </div>
                              <div>
                                <div><br>
                                  <div class="gmail_quote">
                                    <div dir="ltr">Le jeu. 25 août 2016
                                      à 12:17, Francesco Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>>
                                      a écrit :<br>
                                    </div>
                                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                      <div dir="ltr">Thanks a lot!
                                        Pierre I will def will give it a
                                        shot. I am not sure I'll be able
                                        to smash that 'start recording'
                                        button as the UI isusuallystuck
                                        when the CPU spikes. Updates to
                                        come!</div>
                                      <div class="gmail_extra"><br>
                                        <div class="gmail_quote">On Thu,
                                          Aug 25, 2016 at 10:45 AM,
                                          Pierre Tardy <span dir="ltr"><<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>></span>
                                          wrote:<br>
                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                            <div dir="ltr">Hi Francesco,
                                              <div><br>
                                              </div>
                                              <div>I spent some time in
                                                order to implement a
                                                profiler plugin for
                                                buildbot</div>
                                              <div><br>
                                              </div>
                                              <div>You can give it a
                                                look, and send your
                                                profile.json file if you
                                                need more analysis from
                                                me.</div>
                                              <div>
                                                <div><a href="https://github.com/tardyp/buildbot_profiler" target="_blank">https://github.com/tardyp/buildbot_profiler</a><br>
                                                </div>
                                                <div><br>
                                                </div>
                                                <div>Regards,</div>
                                                <div>Pierre</div>
                                                <br>
                                              </div>
                                            </div>
                                            <div>
                                              <div><br>
                                                <div class="gmail_quote">
                                                  <div dir="ltr">Le mer.
                                                    24 août 2016
                                                    à 22:43, Francesco
                                                    Di Mizio <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>>
                                                    a écrit :<br>
                                                  </div>
                                                  <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                    <div dir="ltr">I've
                                                      tried and it's not
                                                      an easy task
                                                      because of my Win
                                                      into Vagrant into
                                                      Docker setup. 
                                                      <div>I'll try
                                                        again soon when
                                                        I get a Linux
                                                         box!</div>
                                                    </div>
                                                    <div class="gmail_extra"><br>
                                                      <div class="gmail_quote">On
                                                        Fri, Aug 19,
                                                        2016 at 5:54 PM,
                                                        Vasily <span dir="ltr"><<a href="mailto:just.one.man@yandex.ru" target="_blank">just.one.man@yandex.ru</a>></span>
                                                        wrote:<br>
                                                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <p dir="ltr">Umm,
                                                          no. VTune has
                                                          Python support
                                                          starting 2017
                                                          Beta, and,
                                                          well, it was
                                                          my team (at
                                                          Intel) work
                                                          actually :-)</p>
                                                          <p dir="ltr">P.S.
                                                          I'm from
                                                          Intel, too. </p>
                                                          <p dir="ltr">Thanks,
                                                          <br>
                                                          Vasily</p>
                                                          <div class="gmail_quote">19
                                                          авг. 2016 г.
                                                          18:17
                                                          пользователь
                                                          "Francesco Di
                                                          Mizio" <<a href="mailto:francescodimizio@gmail.com" target="_blank">francescodimizio@gmail.com</a>>
                                                          написал:
                                                          <div>
                                                          <div><br type="attribution">
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <p dir="ltr">I
                                                          had thought
                                                          you were
                                                          making fun of
                                                          Intel somehow
                                                          ;)</p>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Aug 19, 2016
                                                          5:07 PM,
                                                          "Pierre Tardy"
                                                          <<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>>
                                                          wrote:<br type="attribution">
                                                          <blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">ahah
                                                          <div><br>
                                                          </div>
                                                          <div>I though
                                                          this was a
                                                          taunt on me
                                                          being employed
                                                          by Intel.</div>
                                                          <div>I
                                                          actually had
                                                          mitigated
                                                          experience
                                                          with vtune few
                                                          years ago, and
                                                          didn't know
                                                          they had
                                                          python support
                                                          until then.</div>
                                                          <div>Being an
                                                          opensource
                                                          guy, I usually
                                                          neglegate to
                                                          look at
                                                          proprietary
                                                          stuff.</div>
                                                          <font color="#888888">
                                                          <div><br>
                                                          </div>
                                                          <div>Pierre</div>
                                                          </font></div>
                                                          <div><br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr">Le ven.
                                                          19 août 2016
                                                          à 12:18,
                                                          Vasily <<a href="mailto:just.one.man@yandex.ru" target="_blank">just.one.man@yandex.ru</a>>
                                                          a écrit :<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <p dir="ltr">I'm
                                                          again
                                                          suggesting to
                                                          look into
                                                          Python
                                                          profiling
                                                          capabilities
                                                          of Intel®
                                                          VTune™
                                                          Amplifier. It
                                                          could run
                                                          statistical
                                                          profiling for
                                                          a long time
                                                          and display
                                                          CPU usage over
                                                          time, so the
                                                          developer can
                                                          look at
                                                          specific time
                                                          range where
                                                          CPU usage was
                                                          too high and
                                                          see which
                                                          functions were
                                                          executed. </p>
                                                          <p dir="ltr">Thanks,
                                                          <br>
                                                          Vasily</p>
                                                          <div class="gmail_quote">19
                                                          авг. 2016 г.
                                                          11:57
                                                          пользователь
                                                          "Pierre Tardy"
                                                          <<a href="mailto:tardyp@gmail.com" target="_blank">tardyp@gmail.com</a>>
                                                          написал:</div>
                                                          <div class="gmail_quote"><br type="attribution">
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">Hi
                                                          Francesco,
                                                          <div><br>
                                                          </div>
                                                          <div>Your
                                                          described
                                                          setup looks
                                                          sane to me.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>The
                                                          problems we
                                                          are trying to
                                                          catch are cpu
                                                          spikes, as far
                                                          as I
                                                          understand,
                                                          which does not
                                                          happen for
                                                          very long, but
                                                          are very
                                                          annoying for
                                                          users, as it
                                                          is blocking
                                                          the reactor.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>This
                                                          problem is not
                                                          easy to see in
                                                          the profile
                                                          you sent, as
                                                          this profile
                                                          is over long
                                                          time, so we
                                                          see the
                                                          average of
                                                          each method
                                                          during the day
                                                          and not the
                                                          spikes.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>What
                                                          would really
                                                          be needed is a
                                                          on-demand
                                                          profiler which
                                                          would detect
                                                          cpu spikes and
                                                          only log the
                                                          stack traces
                                                          during those
                                                          times.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Here is a
                                                          nice blog pst
                                                          explaining why
                                                          statistic
                                                          profiling is
                                                          cool and easy
                                                          to implement
                                                          in python.</div>
                                                          <div><a href="https://nylas.com/blog/performance" target="_blank">https://nylas.com/blog/performance</a><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>For 0.9.1
                                                          I want to
                                                          concentrate on
                                                          scalability,
                                                          and write a
                                                          debugging ui
                                                          plugin based
                                                          on those ideas
                                                          (and probably
                                                          code)</div>
                                                          <div><br>
                                                          </div>
                                                          <div>That
                                                          would be great
                                                          if your team
                                                          can help on
                                                          that matter.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Regards,</div>
                                                          <div>Pierre</div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </blockquote>
                                                      </div>
                                                      <br>
                                                    </div>
                                                  </blockquote>
                                                </div>
                                              </div>
                                            </div>
                                          </blockquote>
                                        </div>
                                        <br>
                                      </div>
                                    </blockquote>
                                  </div>
                                </div>
                              </div>
                            </blockquote>
                          </div>
                          <br>
                        </div>
                      </blockquote>
                    </div>
                  </div>
                </div>
              </blockquote>
            </div>
            <br>
          </div>
        </blockquote>
      </div>
      <br>
      <fieldset></fieldset>
      <br>
      </blockquote></div><div bgcolor="#FFFFFF" text="#000000"><blockquote type="cite"><pre>_______________________________________________
users mailing list
<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a>
<a href="https://lists.buildbot.net/mailman/listinfo/users" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a></pre>
    </blockquote></div>

_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a><br>
<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a></blockquote></div>