[Buildbot-commits] [Buildbot] #1992: master hangs on startup

[Buildbot]

#1992: master hangs on startup
------------------------+-------------------------
Reporter:  dustin       |       Owner:  tom.prince
    Type:  defect       |      Status:  assigned
Priority:  major        |   Milestone:  undecided
 Version:  0.8.7        |  Resolution:
Keywords:  performance  |
------------------------+-------------------------

Comment (by mikesorrenti):

 I've seen the possibly-related issue that dank discusses at
 http://sourceforge.net/mailarchive/message.php?msg_id=29899918 on my
 installation. It's intermittent, but ~4/5 attempts to start buildbot
 result in !WebStatus dying with the twistd.log looking like so:

 {{{
 2012-10-19 10:57:29-0700 [-] RotateLogSite starting on 8010
 2012-10-19 10:57:29-0700 [-] Starting factory
 <buildbot.status.web.baseweb.RotateLogSite instance at 0xb12abec>
 2012-10-19 10:57:29-0700 [-] Setting up http.log rotating 10 files of
 10000000 bytes each
 2012-10-19 10:57:29-0700 [-] BuildMaster is running
 2012-10-19 10:57:29-0700 [buildbot.status.web.baseweb.RotateLogSite] (TCP
 Port 8010 Closed)
 2012-10-19 10:57:29-0700 [buildbot.status.web.baseweb.RotateLogSite]
 Stopping factory <buildbot.status.web.baseweb.RotateLogSite instance at
 0xb12abec>
 }}}

 I'm able to replicate this with the provided master.cfg.sample and
 constantly running
 {{{
 buildbot restart master/; sleep 1; tail -n 20 master/twistd.log | grep
 --color "TCP Port 8010 Closed"
 }}}

 The other failure mode I see is closer to the original symptoms discussed
 in this ticket: master just hangs on startup. This occurs much less often
 for me, maybe 1/15.

 System information:
 {{{
 (buildmaster087)buildbot at buildmaster:/usr/local/virtualenvs/buildmaster087$
 pip freeze
 Jinja2==2.6
 PAM==0.4.2
 SQLAlchemy==0.7.9
 Tempita==0.5.1
 Twisted==12.2.0
 Twisted-Conch==10.1.0
 Twisted-Core==10.1.0
 Twisted-Lore==10.1.0
 Twisted-Mail==10.1.0
 Twisted-Names==10.1.0
 Twisted-News==10.1.0
 Twisted-Runner==10.1.0
 Twisted-Web==10.1.0
 Twisted-Words==10.1.0
 buildbot==0.8.7
 buildbot-slave==0.8.7
 decorator==3.3.3
 distribute==0.6.14
 guppy==0.1.10
 iotop==0.4
 numpy==1.4.1
 pyOpenSSL==0.10
 pyasn1==0.0.11a
 pycrypto==2.1.0
 pycryptopp==0.5.17
 pyserial==2.3
 python-apt==0.7.100.1-squeeze1
 python-dateutil==1.5
 reportbug==4.12.6
 sqlalchemy-migrate==0.7.2
 virtualenv==1.7
 wsgiref==0.1.2
 zope.interface==4.0.1
 (buildmaster087)buildbot at buildmaster:/usr/local/virtualenvs/buildmaster087$
 python --version
 Python 2.6.6
 (buildmaster087)buildbot at buildmaster:/usr/local/virtualenvs/buildmaster087$
 uname -a
 Linux buildmaster 2.6.32-rt #1 SMP PREEMPT Tue Jul 19 09:42:50 PDT 2011
 i686 GNU/Linux
 (buildmaster087)buildbot at buildmaster:/usr/local/virtualenvs/buildmaster087$
 cat /etc/debian_version
 6.0.3
 }}}

 It looks like Twisted 12.2.0 isn't a universal fix. I also removed all
 !ForceSchedulers (replaced with a Nightly) and still saw the issue. This
 was in reply to a note dank had made on
 http://sourceforge.net/mailarchive/message.php?msg_id=29899918

    my buildbot master gets part of the way through startup, then hangs on
 adding the first scheduler (a force scheduler, hmm!)

-- 
Ticket URL: <http://trac.buildbot.net/ticket/1992#comment:19>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation