<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Pierre,<br>
<br>
Thanks for the specifics. If it comes up again, I'll have a look in
the places you describe.<br>
<br>
It's slightly possible that this might have happened during a
reconfiguration, though unlikely. I'm stll trying to train the users
to use my supplied script to reconfigure, as it does a checkconfig
and reconfigures all 4 masters.<br>
<br>
I think a problem during the health-check period is much more
likely. Though none of the masters crashed, it seems more likely
that some network problem might have caused it to appear so. Next,
time, I'll try the reconfig first. In this case, restarting wasn't a
problem. If nothing else, it corrected some of the
non-reconfigurable scheduler troubles, if only temporarily.<br>
<br>
Neil Gilmore<br>
grammatech.com<br>
<br>
<div class="moz-cite-prefix">On 12/8/2016 2:25 PM, Pierre Tardy
wrote:<br>
</div>
<blockquote
cite="mid:CAJ+soVfRxrXJEZ4Q9MGxF5Xa5VK2MkOj681KZoD1BdnTxKj2Fg@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Neil,
<div>Thanks for the detailed report. I see few chances that the
symptoms you are describing could be explained by a failure of
the multi-master messaging.</div>
<div>If the data api showed the builders that means that the
builders were seen to be attached to 0 masters.</div>
<div>There is a "show old builders" checkbox that could have
confirm that. An builder is considered "old" when it has no
master.</div>
<div>The builders REST api has a masterids attribute that will
tell that.</div>
<div><br>
</div>
<div>There are several action that will make the list of
builders of a given master go to 0</div>
<div><br>
</div>
<div>- during a reconfiguration. The BotMaster service will
setup the new list of builders to the database (they could go
to 0 if misconfiguration)</div>
<div><br>
</div>
<div>- at master shutdown, the master will set itself inactive,
and unregisters from all its builders.</div>
<div><br>
</div>
<div>- After the master health-check period. each master has a
timestamp which a needs to update regularly in the database to
inform other masters that he his still alive. During that
heartbeat callback, the master will also check for other
masters if they have correctly updated their own timestamp. If
they didn't for the previous 10 minutes, this means that they
somehow crashed without telling, so the first detecting master
will mark the quiet master to be disconnected. In you case,
this could be explaining the behaviour. Maybe there was a time
were the consumer and procucers masters were unavailable,
blocked or off-network. The third master marked them away, but
the 2 then went back, but did not figure out they were marked
disconnected, but still continued to take buildrequests. I
think this is a design bug that we need to fixed. A single
reconfig would have fixed the situation (no need for restart)</div>
<div><br>
</div>
<div>In any case I would expect that the twisted.log may tell
you some stuff. Either you would get some exceptions during a
reconfiguration or something. Or you may get a period of time
with suspicious activity, which could explain a miss of the
heartbeat timer.<br>
<br>
Let us know if you reproduce the problem again and if these
advices helped you better understand the problem.</div>
<div>regards,</div>
<div>Pierre</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">Le jeu. 8 déc. 2016 à 17:45, Neil Gilmore
<<a moz-do-not-send="true"
href="mailto:ngilmore@grammatech.com">ngilmore@grammatech.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi
everyone.<br class="gmail_msg">
<br class="gmail_msg">
First, a bit of good news. My current top priority is to
make the<br class="gmail_msg">
schedulers reconfigurable. Not conceptually difficult, but
I wasn't<br class="gmail_msg">
well-versed in Python argument passing (which figures
prominently in<br class="gmail_msg">
this), so I've had a couple aborted tries on that score. I
think I've<br class="gmail_msg">
got all that sorted out for now. It's just biting us way
too badly to<br class="gmail_msg">
not be able to reconfigure schedulers.<br
class="gmail_msg">
<br class="gmail_msg">
Now, the anecdote. As you may remember, we're running 4
masters. 1 just<br class="gmail_msg">
has the UI and force schedulers. 1 has our overall logging
system. The<br class="gmail_msg">
other 2 are split between producing builds, and consuming
them for tests.<br class="gmail_msg">
<br class="gmail_msg">
Sometime between when I left yesterday and when the test
lead looked<br class="gmail_msg">
this morning, the UI stopped displaying the builders for
the producer<br class="gmail_msg">
and consumer masters. Looking at all the masters, they
were running, and<br class="gmail_msg">
I didn't immediately see anything suspicious in the logs.
Looking at the<br class="gmail_msg">
data api, I could see all the builders and workers. The
workers all<br class="gmail_msg">
showed connected_to being valid, but only the logging
workers showed<br class="gmail_msg">
anything in configured_on. I restarted our UI master and
that didn't<br class="gmail_msg">
help. Restarting the producer and consumer seems to have
solved the<br class="gmail_msg">
problem. I can see the builders in the UI, and looking at
the workers in<br class="gmail_msg">
the data API, I see that most appear to have configured_on
set. I have<br class="gmail_msg">
no idea what actually happened. My wild conjecture is that
the<br class="gmail_msg">
inter-master communication got screwed up somehow. Either
that or they<br class="gmail_msg">
lost connection to the database (less likely, I think.
Postgres is<br class="gmail_msg">
pretty stable that way.).<br class="gmail_msg">
<br class="gmail_msg">
Neil Gilmore<br class="gmail_msg">
<a moz-do-not-send="true" href="http://grammatech.com"
rel="noreferrer" class="gmail_msg" target="_blank">grammatech.com</a><br
class="gmail_msg">
_______________________________________________<br
class="gmail_msg">
users mailing list<br class="gmail_msg">
<a moz-do-not-send="true" href="mailto:users@buildbot.net"
class="gmail_msg" target="_blank">users@buildbot.net</a><br
class="gmail_msg">
<a moz-do-not-send="true"
href="https://lists.buildbot.net/mailman/listinfo/users"
rel="noreferrer" class="gmail_msg" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br
class="gmail_msg">
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>