<div dir="ltr">Sorry, I realize now my answer was a bit too direct. I was answering from my phone.<div><br></div><div>I do care about Buildbot hanging, this is something we should resolve. I dont see anything obvious from the master.cfg you provided.</div><div>So we really need to dig further together.</div><div><br></div><div>It looks like you are only using one worker, which is probably on the same machine.</div><div>In that case, I would advise you to configure LocalWorker instead of Worker, which avoids you to manage a separate process.</div><div><a href="http://docs.buildbot.net/latest/manual/cfg-workers.html#local-workers">http://docs.buildbot.net/latest/manual/cfg-workers.html#local-workers</a><br></div><div><br></div><div>Regards</div><div>Pierre</div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Apr 6, 2017 at 7:57 PM Pierre Tardy <<a href="mailto:tardyp@gmail.com">tardyp@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr" class="gmail_msg">There is no concept of "pending build" in buildbot nine UI anymore. Are you seeing buildrequests or builds in the builder UI?</p>
<p dir="ltr" class="gmail_msg">Also the logs are from forth of April and we are the six. Are you sure you are looking at the right log?<br class="gmail_msg">
You will need to look at the last meaningful message which is not poller or keep alive. It is very often that the answer is in the log, but the log is full of noise and I have hard time finding out. <br class="gmail_msg">
</p>
<br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">Le jeu. 6 avr. 2017 19:37, Chris Spencer <<a href="mailto:chrisspen@gmail.com" class="gmail_msg" target="_blank">chrisspen@gmail.com</a>> a écrit :<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg">Buildbot shows pending builds, but never begins running the build, almost like the worker has crashed. However, the worker's twistd.log shows it's still running.<br class="gmail_msg"><br class="gmail_msg"></div>I also thought that my experimental nextBuild callback was the cause of the issue, so I removed it and restarted buildbot. However, the problem existed before I added my callback, and the problem remains after I removed it.<br class="gmail_msg"></div><div class="gmail_extra gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg">On Thu, Apr 6, 2017 at 1:19 PM, Pierre Tardy <span dir="ltr" class="gmail_msg"><<a href="mailto:tardyp@gmail.com" class="gmail_msg" target="_blank">tardyp@gmail.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr" class="gmail_msg">Hi Chris. I am not sure exactly what you mean as hang. Could you please describe me what happens and what should happen instead for you?</p>
<p dir="ltr" class="gmail_msg">I see you are also playing with nextBuild in another thread. Beware that a crashing or misbehaving nextBuild can easily make buildbot stop creating new build.<br class="gmail_msg">
</p>
<br class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><div class="m_3776804910960453271m_-1800899228125902416h5 gmail_msg"><div dir="ltr" class="gmail_msg">Le jeu. 6 avr. 2017 16:44, Chris Spencer <<a href="mailto:chrisspen@gmail.com" class="gmail_msg" target="_blank">chrisspen@gmail.com</a>> a écrit :<br class="gmail_msg"></div></div></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_msg"><div class="m_3776804910960453271m_-1800899228125902416h5 gmail_msg"><div dir="ltr" class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"><div class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"><div class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">OS: Ubuntu 16.04<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"></div>Twisted: 17.1.0<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"></div>Buildbot: 0.9.4<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"><br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"><div class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"><br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"></div></div><div class="gmail_extra m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"><br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"><div class="gmail_quote m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">On Thu, Apr 6, 2017 at 10:29 AM, Dan Kegel <span dir="ltr" class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"><<a href="mailto:dank@kegel.com" class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg" target="_blank">dank@kegel.com</a>></span> wrote:<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"><blockquote class="gmail_quote m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">On Thu, Apr 6, 2017 at 6:54 AM, Chris Spencer <<a href="mailto:chrisspen@gmail.com" class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg" target="_blank">chrisspen@gmail.com</a>> wrote:<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
> I'm continuing to see some strange issues with Buildbot hanging. I tried<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
> using a crontab to see if the worker or master had crashed and restart them,<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
> but it looks like they're still running just fine. Buildbot's even showing<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
> pending builds, but it's not ignoring them.<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
><br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
> If I manually restart them, it immediately begins working again, but I have<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
> no way to detect it being hung except to login and check if there are<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
> pending builds with nothing being processed.<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
><br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
> How do I diagnose and fix this?<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
</span>I know the pain. (On cygwin, with a very old twisted, I'm<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
seeing a hang very often at the end of the git step; twisted<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
is failing to notice its child has finished. I have also seen<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
hangs on Linux, but not that one, and much less often.)<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
What OS, and versions of twisted and buildbot?<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
Can you get a backtrace from the hung worker?<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
</blockquote></div><br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg"></div></div></div>
_______________________________________________<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
users mailing list<br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
<a href="mailto:users@buildbot.net" class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg" target="_blank">users@buildbot.net</a><br class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg">
<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" class="m_3776804910960453271m_-1800899228125902416m_-3939228543352312762gmail_msg gmail_msg" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a></blockquote></div>
</blockquote></div><br class="gmail_msg"></div>
</blockquote></div></blockquote></div>