<div dir="ltr"><div><div><div>I'm continuing to see some strange issues with Buildbot hanging. I tried using a crontab to see if the worker or master had crashed and restart them, but it looks like they're still running just fine. Buildbot's even showing pending builds, but it's not ignoring them.<br><br></div><div>If I manually restart them, it immediately begins working again, but I have no way to detect it being hung except to login and check if there are pending builds with nothing being processed.<br></div><div><br></div><div>How do I diagnose and fix this?<br></div><div><br></div>My master's log is just a bunch of:<br><br>    2017-04-04 18:15:41-0400 [-] gitpoller: processing changes from "git@bitbucket.org:myorg/<wbr>myproject.git"<br>    2017-04-04 18:16:41-0400 [-] gitpoller: processing changes from "git@bitbucket.org:myorg/<wbr>myproject.git"<br>    2017-04-04 18:17:41-0400 [-] gitpoller: processing changes from "git@bitbucket.org:myorg/<wbr>myproject.git"<br>    2017-04-04 18:18:41-0400 [-] gitpoller: processing changes from "git@bitbucket.org:myorg/<wbr>myproject.git"<br><br></div>and my worker's log is just a bunch of:<br><br>    2017-04-04 16:13:02-0400 [-] sending app-level keepalive<br>    2017-04-04 16:23:02-0400 [-] sending app-level keepalive<br>    2017-04-04 16:33:02-0400 [-] sending app-level keepalive<br>    2017-04-04 16:43:02-0400 [-] sending app-level keepalive<br>    2017-04-04 16:53:02-0400 [-] sending app-level keepalive<br>    2017-04-04 17:03:03-0400 [-] sending app-level keepalive<br>    2017-04-04 17:13:03-0400 [-] sending app-level keepalive<br><br></div>This is my master.cfg:<br><br>    # -*- python -*-<br>    # ex: set syntax=python:<br>    import os<br><br>    from buildbot.plugins import util, reporters, worker, changes, schedulers, steps<br>    from buildbot.reporters.bitbucket import BitbucketStatusPush<br><br>    from bitbucket_sensitive import BITBUCKET_OATH_KEY, BITBUCKET_OATH_SECRET # pylint: disable=import-error,no-name-in-module<br>    import master_sensitive # pylint: disable=import-error<br><br>    BASE_DIR = os.path.abspath(os.path.dirname(__file__))<br><br>    HOME_DIR = '/var/lib/buildbot'<br><br>    c = BuildmasterConfig = {}<br><br>    c['workers'] = [worker.Worker("example-worker", "pass")]<br>    c['protocols'] = {'pb': {'port': 9989}}<br>    c['change_source'] = []<br>    c['change_source'].append(changes.GitPoller(<br>            'git@bitbucket.org:myorg/myproject.git',<br>            workdir='gitpoller-workdir',<br>            branches=True,<br>            pollinterval=60, # interval in seconds between polls<br>    ))<br><br>    c['schedulers'] = []<br>    c['schedulers'].append(schedulers.AnyBranchScheduler(<br>        name="all",<br>        treeStableTimer=60,<br>        builderNames=["myproject_runtests"],<br>    ))<br>    c['schedulers'].append(schedulers.ForceScheduler(<br>        name="force",<br>        builderNames=["myproject_runtests"]))<br><br>    factory = util.BuildFactory()<br>    factory.addStep(steps.Git(<br>        repourl='git@bitbucket.org:myorg/myproject.git',<br>        mode='incremental',<br>    ))<br>    factory.addStep(steps.ShellCommand(<br>        command="rm -rf $(find . -name '*.pyc' 2>/dev/null)",<br>        workdir="/usr/local/myproject",<br>        haltOnFailure=True,<br>        env={'HOME': HOME_DIR},<br>    ))<br>    factory.addStep(steps.ShellCommand(<br>        command=['git', 'fetch', 'origin', 'master'],<br>        workdir="build",<br>        haltOnFailure=True,<br>        env={'HOME': HOME_DIR},<br>    ))<br>    factory.addStep(steps.ShellCommand(<br>        command=['git', 'merge', 'origin/master', '--no-edit'],<br>        workdir="build",<br>        haltOnFailure=True,<br>        env={'HOME': HOME_DIR},<br>    ))<br>    factory.addStep(steps.ShellCommand(<br>        command=['git', 'merge', 'origin/staging', '--no-edit'],<br>        workdir="build",<br>        haltOnFailure=True,<br>        env={'HOME': HOME_DIR},<br>    ))<br>    factory.addStep(steps.ShellCommand(<br>        command=['./init_virtualenv.sh'],<br>        workdir="build/src/scripts",<br>        haltOnFailure=True,<br>        env={'HOME': HOME_DIR},<br>    ))<br>    factory.addStep(steps.ShellCommand(<br>        command=['./checkjs.sh'],<br>        workdir="build",<br>        haltOnFailure=True,<br>        env={'HOME': HOME_DIR},<br>    ))<br>    factory.addStep(steps.ShellCommand(<br>        command=['./pep8.sh'],<br>        workdir="build",<br>        haltOnFailure=True,<br>        env={'HOME': HOME_DIR},<br>    ))<br>    factory.addStep(steps.ShellCommand(<br>        command=["./run_buildbot_tests"],<br>        workdir="build/src/myproject",<br>        env={'HOME': HOME_DIR},<br>    ))<br><br>    c['builders'] = []<br>    c['builders'].append(<br>        util.BuilderConfig(<br>            name="myproject_runtests",<br>            workernames=["example-worker"],<br>            collapseRequests=True,<br>            factory=factory))<br><br>    c['status'] = []<br><br>    authz = util.Authz(<br>      allowRules=[<br>        util.StopBuildEndpointMatcher(role="admins"),<br>        util.ForceBuildEndpointMatcher(role="admins"),<br>        util.RebuildBuildEndpointMatcher(role="admins")<br>      ],<br>      roleMatchers=[<br>        util.RolesFromEmails(admins=["<a href="mailto:myuser@myorg.com">myuser@myorg.com</a>"]),<br>        util.RolesFromUsername(roles=["admins"], usernames=["myuser"]),<br>      ]<br>    )<br>    c['www'] = dict(<br>        port=8010,<br>        plugins=dict(waterfall_view={}, console_view={}),<br>        #auth=util.HTPasswdAuth(os.path.join(BASE_DIR, '../htpasswd')),<br>        auth=util.RemoteUserAuth(),<br>        authz=authz,<br>    )<br><br>    mn = reporters.MailNotifier(<br>        fromaddr="<a href="mailto:buildbot@myproject.com">buildbot@myproject.com</a>",<br>        sendToInterestedUsers=True,<br>        useTls=True,<br>        relayhost="<a href="http://smtp.myproject.com">smtp.myproject.com</a>",<br>        smtpPort=587,<br>        smtpUser="<a href="mailto:buildbot@myproject.com">buildbot@myproject.com</a>",<br>        smtpPassword=master_sensitive.EMAIL_HOST_PASSWORD,<br>        extraRecipients=[<br>            '<a href="mailto:developers@myproject.com">developers@myproject.com</a>',<br>        ],<br>    )<br>    c['services'] = []<br>    c['services'].append(mn)<br><br>    bs = BitbucketStatusPush(BITBUCKET_OATH_KEY, BITBUCKET_OATH_SECRET)<br>    c['services'].append(bs)<br><br>    c['title'] = "MyProject"<br><br>    c['buildbotURL'] = "<a href="http://buildbot.myproject.com/">http://buildbot.myproject.com/</a>"<br><br>    c['db'] = {<br>        'db_url' : "sqlite:///state.sqlite",<br>    }<br><br>    c['changeHorizon'] = 200<br>    c['buildHorizon'] = 100<br>    c['logHorizon'] = 40<br>    c['buildCacheSize'] = 15<br><br></div>