[users at bb.net] Buildbot worker periodically stopping

[Pierre Tardy]

SIGTERM at exactly 2AM. This sounds like a cronjob.

The connection lost message is related to the sigterm.

Hope this helps,
Pierre

Le jeu. 23 mars 2017 à 21:47, Chris Spencer <chrisspen at gmail.com> a écrit :

> I'm noticing my Buildbot worker seems to be randomly stopping, causing all
> builds to stop.
>
> The end of my worker's log shows:
>
>     [buildbot-server] out: 2017-03-22 19:10:55-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 19:20:55-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 19:30:56-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 19:40:56-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 19:50:56-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 20:00:55-0400
> [HangCheckProtocol,client] message from master: keepalive
>     [buildbot-server] out: 2017-03-22 20:00:56-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 20:10:56-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 20:20:56-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 20:30:56-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 20:40:56-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 20:50:56-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 21:00:56-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 21:10:56-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 21:20:57-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 21:30:57-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 21:40:57-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 21:50:57-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 22:00:57-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 22:10:57-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 22:20:57-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 22:30:57-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 22:40:57-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 22:50:57-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 23:00:58-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 23:10:58-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 23:20:58-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 23:30:58-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 23:40:58-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-22 23:50:58-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 00:00:58-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 00:10:58-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 00:20:58-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 00:30:58-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 00:40:58-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 00:50:59-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 01:00:59-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 01:10:59-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 01:20:59-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 01:30:59-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 01:40:59-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 01:50:59-0400 [-] sending app-level
> keepalive
>     [buildbot-server] out: 2017-03-23 02:00:00-0400 [-] Received SIGTERM,
> shutting down.
>     [buildbot-server] out: 2017-03-23 02:00:00-0400
> [HangCheckProtocol,client] Lost connection to localhost:9989
>     [buildbot-server] out: 2017-03-23 02:00:00-0400 [-] Stopping factory
> <buildbot_worker.pb.BotFactory instance at 0x7fdca6b3ccf8>
>     [buildbot-server] out: 2017-03-23 02:00:00-0400 [-] Main loop
> terminated.
>     [buildbot-server] out: 2017-03-23 02:00:00-0400 [-] Server Shut Down.
>
> What's causing this? What would have sent the worker a SIGTERM? The worker
> and master are on the same server, so I'm not sure what would be causing a
> "Lost connection" error.
>
> Master still appears to be running, as I can still access the web UI just
> fine.
>
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20170323/fb656823/attachment.html>