[Buildbot-commits] [Buildbot] #2753: master does not reconnect to PostgreSQL when database server is restarted

[Buildbot trac]

#2753: master does not reconnect to PostgreSQL when database server is restarted
------------------------+-----------------------
Reporter:  mstapelberg  |      Owner:
    Type:  undecided    |     Status:  new
Priority:  major        |  Milestone:  undecided
 Version:  0.8.8        |   Keywords:
------------------------+-----------------------
 On every PostgreSQL upgrade, I need to restart buildbot. I don’t think
 this should be necessary — instead, buildbot should simply reconnect :).

 Even worse is that the failure mode is pretty horrible. The
 git_buildbot.py script doesn’t output anything and you simply don’t get
 any new builds.

 Here is a stack trace of the problem:

 {{{
 2014-04-09 22:19:48+0200 [Broker,7,79.140.39.201] perspective_addChange
 called
 2014-04-09 22:19:48+0200 [-] Peer will receive following PB traceback:
 2014-04-09 22:19:48+0200 [-] Unhandled Error
         Traceback (most recent call last):
           File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
             self.__bootstrap_inner()
           File "/usr/lib/python2.7/threading.py", line 810, in
 __bootstrap_inner
             self.run()
           File "/usr/lib/python2.7/threading.py", line 763, in run
             self.__target(*self.__args, **self.__kwargs)
         --- <exception caught here> ---
           File "/usr/lib/python2.7/dist-
 packages/twisted/python/threadpool.py", line 191, in _worker
             result = context.call(ctx, function, *args, **kwargs)
           File "/usr/lib/python2.7/dist-
 packages/twisted/python/context.py", line 118, in callWithContext
             return self.currentContext().callWithContext(ctx, func, *args,
 **kw)
           File "/usr/lib/python2.7/dist-
 packages/twisted/python/context.py", line 81, in callWithContext
             return func(*args,**kw)
           File "/usr/lib/python2.7/dist-packages/buildbot/db/pool.py",
 line 184, in __thd
             rv = callable(arg, *args, **kwargs)
           File "/usr/lib/python2.7/dist-packages/buildbot/db/users.py",
 line 40, in thd
             rows = conn.execute(q).fetchall()
           File "/usr/lib/python2.7/dist-
 packages/sqlalchemy/engine/base.py", line 720, in execute
             return meth(self, multiparams, params)
           File "/usr/lib/python2.7/dist-
 packages/sqlalchemy/sql/elements.py", line 317, in _execute_on_connection
             return connection._execute_clauseelement(self, multiparams,
 params)
           File "/usr/lib/python2.7/dist-
 packages/sqlalchemy/engine/base.py", line 817, in _execute_clauseelement
             compiled_sql, distilled_params
           File "/usr/lib/python2.7/dist-
 packages/sqlalchemy/engine/base.py", line 947, in _execute_context
             context)
           File "/usr/lib/python2.7/dist-
 packages/sqlalchemy/engine/base.py", line 1108, in _handle_dbapi_exception
             exc_info
           File "/usr/lib/python2.7/dist-
 packages/sqlalchemy/util/compat.py", line 185, in raise_from_cause
             reraise(type(exception), exception, tb=exc_tb)
           File "/usr/lib/python2.7/dist-
 packages/sqlalchemy/engine/base.py", line 940, in _execute_context
             context)
           File "/usr/lib/python2.7/dist-
 packages/sqlalchemy/engine/default.py", line 435, in do_execute
             cursor.execute(statement, parameters)
         sqlalchemy.exc.OperationalError: (OperationalError) SSL connection
 has been closed unexpectedly
          'SELECT users_info.uid \nFROM users_info \nWHERE
 users_info.attr_type = %(attr_type_1)s AND users_info.attr_data =
 %(attr_data_1)s' {'attr_data_1': u'Michael Stapelberg
 <michael at example.com>', 'attr_type_1': u'git'}

 }}}

 Let me know if you need any other information. Thanks.

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2753>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation