<div dir="ltr">Hi,<div><br></div><div>Thanks for reporting. I am not sure if your issue is related to the TLS hack.</div><div>Please note that we now have official TLS support in mainline.</div><div><br></div><div>We do have tests which make sure the connection is restored after a network break down, but there are so many different way a TCP connection can break that it is difficult to test.</div><div>We would be glad if you can test the new TLS support regarding this.</div><div><br></div><div>In any way, the best for such issue is to raise a bug on github</div><div><br></div><div>Regards</div><div>PIerre </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Mar 2, 2019 at 3:33 PM Yngve N. Pettersen <<a href="mailto:yngve@vivaldi.com">yngve@vivaldi.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello all,<br>
<br>
Yesterday we had a network event when some of our buildbot workers lost <br>
the network connection to the master for about 10 minutes.<br>
<br>
However, while according to the logs on both the master and the workers <br>
show that the workers successfully reconnected within 10 minutes of the <br>
network connection being restored, according to status displays, the <br>
workers the worker were missing. It eventually took a stop/start or reboot <br>
of the workers to get them reconnected an hour after the network <br>
connection was lost.<br>
<br>
What I am seeing is that master log has entries like this when a worker <br>
("arbeider") reconnected:<br>
<br>
2019-03-01 11:24:05+0000 [Broker (TLSMemoryBIOProtocol),236,1.2.3.4] <br>
worker 'arbeider' attaching from IPv4Address(type='TCP', host='1.2.3.4', <br>
port=51630)<br>
2019-03-01 11:24:05+0000 [Broker (TLSMemoryBIOProtocol),236,1.2.3.4] Got <br>
duplication connection from 'arbeider' starting arbitration procedure<br>
2019-03-01 11:24:15+0000 [-] Connected worker 'arbeider' ping timed out <br>
after 10 seconds<br>
2019-03-01 11:24:15+0000 [-] Old connection for 'arbeider' was lost, <br>
accepting new<br>
2019-03-01 11:24:15+0000 [Broker (TLSMemoryBIOProtocol),236,1.2.3.4] Got <br>
workerinfo from 'arbeider'<br>
2019-03-01 11:24:15+0000 [Broker (TLSMemoryBIOProtocol),236,1.2.3.4] <br>
worker arbeider cannot attach<br>
Traceback (most recent call last):<br>
File <br>
"sandbox/lib/python3.6/site-packages/twisted/internet/defer.py", line <br>
1529, in _cancellableInlineCallbacks<br>
_inlineCallbacks(None, g, status)<br>
File <br>
"sandbox/lib/python3.6/site-packages/twisted/internet/defer.py", line <br>
1416, in _inlineCallbacks<br>
result = result.throwExceptionIntoGenerator(g)<br>
File <br>
"sandbox/lib/python3.6/site-packages/twisted/python/failure.py", line 491, <br>
in throwExceptionIntoGenerator<br>
return g.throw(self.type, self.value, self.tb)<br>
File <br>
"sandbox/lib/python3.6/site-packages/buildbot/worker/base.py", line 638, <br>
in attached<br>
log.err(e, "worker %s cannot attach" % (<a href="http://self.name" rel="noreferrer" target="_blank">self.name</a>,))<br>
--- <exception caught here> ---<br>
File <br>
"sandbox/lib/python3.6/site-packages/buildbot/worker/base.py", line 636, <br>
in attached<br>
yield AbstractWorker.attached(self, bot)<br>
builtins.AssertionError:<br>
<br>
<br>
Does anyone have any ideas about why the reconnects failed?<br>
<br>
In one case, a job was started on one of the workers (which was shown as <br>
"online"), and the master was just registering the task as "Pinging <br>
worker", for 20+ minutes until we stopped the task (and even that took a <br>
while).<br>
<br>
If this happens every time the network connection is lost (which <br>
admittedly does not happen that frequently, but could happen in case of <br>
network maintenance) it is going to be a serious inconvenience, since some <br>
of the workers need special handling when being restarted.<br>
<br>
<br>
Relevant information about the configuration:<br>
<br>
* Buildbot v2.0.1<br>
<br>
* The PB connections are TLS protected, using a workaround based on the <br>
one from <<a href="https://github.com/buildbot/buildbot/issues/2866" rel="noreferrer" target="_blank">https://github.com/buildbot/buildbot/issues/2866</a>><br>
<br>
* Workers run Python 2<br>
<br>
* The master is running the current Twisted version<br>
<br>
* The workers are running Twisted 18.7.0 (fixed version, due to <br>
installation problems with the current version; on Windows it goes looking <br>
for a compiler and does not find one, even when one is installed)<br>
<br>
<br>
-- <br>
Sincerely,<br>
Yngve N. Pettersen<br>
Vivaldi Technologies AS<br>
_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a><br>
<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br>
</blockquote></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><p dir="ltr"><br></p>
</div></div>