<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Pierre,<br>
<br>
We've had notify_on_missing configured for some time -- certainly it
was present when I restarted the masters on Tues. I also restarted
the master that has those workers. Shouldn't all the workers in
master.cfg have attached then?<br>
<br>
This could cause us some trouble. Here's how we start workers:<br>
<br>
We have a builder consisting of ShellCommands that does an ssh login
into the worker machine, sees if the worker is running, and runs it
if it is not. This allows us to make certain that all the workers
that are in master.cfg also have matching worker processes on the
worker machines. This builder runs every hour. It is entirely
possible that a worker machine without a worker process could sit
for more than an hour before having its worker process started.<br>
<br>
What you guys seem to be telling me is that if I were to stop a
worker process and let more than an hour go by, that worker would
never, ever have its builds run. Even though the worker is attached,
and the builds are queued.<br>
<br>
That sounds pretty bad to me. Am I understanding correctly? Or can I
wait until a worker process is running, then reconfigure without
that worker in master.cfg(and getting unkown worker errors, I
suppose), then reconfiguring with the worker back in master.cfg, so
that it attempts to attach before the timeout?<br>
<br>
Let me emphasize that the thing that brought this to my attention
was adding a new worker and its builders to master.cfg. The worker
process would have been started sometime after that by the builder
that starts worker processes.<br>
<br>
Neil Gilmore<br>
grammatech.com<br>
<br>
<div class="moz-cite-prefix">On 2/3/2017 3:17 PM, Pierre Tardy
wrote:<br>
</div>
<blockquote
cite="mid:CAJ+soVcGuyqyfGonWrpTbrpbSDU1NM-XCy8Vd9M7=5Rr7trO=w@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Neil,
<div><br>
</div>
<div>The timer starts when the worker is first configured.</div>
<div>but only if notify_on_missing is configured.</div>
<div><br>
</div>
<div>that may be a reason why you do not see the bug for ancient
workers</div>
<div><br>
</div>
<div>Pierre</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Le ven. 3 févr. 2017 à 21:59, Neil Gilmore <<a
moz-do-not-send="true" href="mailto:ngilmore@grammatech.com">ngilmore@grammatech.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Andrej,<br
class="gmail_msg">
<br class="gmail_msg">
Thanks for the reply.<br class="gmail_msg">
<br class="gmail_msg">
I don't see missing_timeout in our master.cfg anywhere. But I
do see this:<br class="gmail_msg">
<br class="gmail_msg">
c['workers'] = [Worker(host, '<password>',<br
class="gmail_msg">
notify_on_missing=bots_email[host]) for host in bots_list]<br
class="gmail_msg">
<br class="gmail_msg">
Let's see if I understood you. The default missing_timeout is
60<br class="gmail_msg">
minutes. If I start the master and wait 60 minutes, then start
the<br class="gmail_msg">
worker, the worker won't attach?<br class="gmail_msg">
<br class="gmail_msg">
In our case, we're not even adding the worker to master.cfg
until well<br class="gmail_msg">
after that 60 minutes (a couple days after). We're adding new
workers.<br class="gmail_msg">
Do you figure this could be the same problem?<br
class="gmail_msg">
<br class="gmail_msg">
What happens with a default notify_on_missing? I figure I can
try the<br class="gmail_msg">
patch in your PR when we restart the masters.<br
class="gmail_msg">
<br class="gmail_msg">
Neil Gilmore<br class="gmail_msg">
<a moz-do-not-send="true" href="mailto:raito@raito.com"
class="gmail_msg" target="_blank">raito@raito.com</a><br
class="gmail_msg">
<br class="gmail_msg">
On 2/3/2017 2:42 PM, Andrej Rode wrote:<br class="gmail_msg">
> Hi Neil,<br class="gmail_msg">
><br class="gmail_msg">
>> 2017-02-03T12:39:09-0500 [Broker,28906,10.233.216.43]
worker '<name>'<br class="gmail_msg">
>> attaching from IPv4Address(TCP, '<ip>', 35642)<br
class="gmail_msg">
>> 2017-02-03T12:39:09-0500 [Broker,28906,10.233.216.43]
Got workerinfo<br class="gmail_msg">
>> from '<name>'<br class="gmail_msg">
>> 2017-02-03T12:39:09-0500 [-] bot attached<br
class="gmail_msg">
>> 2017-02-03T12:39:09-0500 [-] worker <name>
cannot attach<br class="gmail_msg">
>> Traceback (most recent call last):<br
class="gmail_msg">
>> Failure:
twisted.internet.error.AlreadyCalled: Tried to cancel<br
class="gmail_msg">
>> an already-called event.<br class="gmail_msg">
> I had the same problembs but with a single-master setup.
By any chance<br class="gmail_msg">
> are you using a non-default `missing_timeout` and/or
`notify_on_missing`<br class="gmail_msg">
> on your workers?<br class="gmail_msg">
><br class="gmail_msg">
> For my issue I've a PR up [0] and now I can detach and
attach workers<br class="gmail_msg">
> as I like. But it is still not clear why we even run into
problems here.<br class="gmail_msg">
><br class="gmail_msg">
> I figured out that attaching a worker after longer than<br
class="gmail_msg">
> `missing_timeout` after a master start results in this
problem on my<br class="gmail_msg">
> setup. (Default `missing_timeout` is 60 minutes.)<br
class="gmail_msg">
><br class="gmail_msg">
> Cheers,<br class="gmail_msg">
> Andrej<br class="gmail_msg">
><br class="gmail_msg">
> [0] <a moz-do-not-send="true"
href="https://github.com/buildbot/buildbot/pull/2708"
rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/buildbot/buildbot/pull/2708</a><br
class="gmail_msg">
> _______________________________________________<br
class="gmail_msg">
> users mailing list<br class="gmail_msg">
> <a moz-do-not-send="true"
href="mailto:users@buildbot.net" class="gmail_msg"
target="_blank">users@buildbot.net</a><br class="gmail_msg">
> <a moz-do-not-send="true"
href="https://lists.buildbot.net/mailman/listinfo/users"
rel="noreferrer" class="gmail_msg" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br
class="gmail_msg">
<br class="gmail_msg">
_______________________________________________<br
class="gmail_msg">
users mailing list<br class="gmail_msg">
<a moz-do-not-send="true" href="mailto:users@buildbot.net"
class="gmail_msg" target="_blank">users@buildbot.net</a><br
class="gmail_msg">
<a moz-do-not-send="true"
href="https://lists.buildbot.net/mailman/listinfo/users"
rel="noreferrer" class="gmail_msg" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br
class="gmail_msg">
</blockquote>
</div>
</blockquote>
<br>
</body>
</html>