[Buildbot] #3593: GitPoller crashes when the repository has too many branches (MySQL Truncates JSON).
Buildbot trac
trac at buildbot.net
Tue Aug 9 16:25:53 UTC 2016
#3593: GitPoller crashes when the repository has too many branches (MySQL
Truncates JSON).
---------------------+--------------------
Reporter: rayene | Owner:
Type: defect | Status: new
Priority: critical | Milestone: 0.9.+
Version: 0.9.0b9 | Resolution:
Keywords: db |
---------------------+--------------------
Changes (by dustin):
* keywords: => db
* milestone: undecided => 0.9.+
Old description:
> GitPoller stores a json value in the object_state MySQL table.
> The type of value_json field is TEXT. TEXT is limited to 65,535 bytes.
>
> When many branches with long names exist in the git repo, the json string
> generated by GitPoller can be larger than 65,535 bytes. MySQL truncates
> the string.
>
> When GitPoller tries to read back the value, it does not find a well-
> formed json string.
>
> Exception :
>
> 2016-08-05 18:33:49+0000 [-] Got fatal Exception on DB
> Traceback (most recent call last):
> Failure: exceptions.TypeError: JSON error loading state value
> 'lastRev' for 2
>
> 2016-08-05 18:33:49+0000 [-] Starting factory
> <twisted.web.client._HTTP11ClientFactory instance at 0x7f5bbd3f2cb0>
> 2016-08-05 18:33:49+0000 [-] while initializing GitPoller repository
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-
> packages/twisted/internet/defer.py", line 434, in errback
> self._startRunCallbacks(fail)
> File "/usr/local/lib/python2.7/dist-
> packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
> self._runCallbacks()
> File "/usr/local/lib/python2.7/dist-
> packages/twisted/internet/defer.py", line 588, in _runCallbacks
> current.result = callback(current.result, *args, **kw)
> File "/usr/local/lib/python2.7/dist-
> packages/twisted/internet/defer.py", line 1184, in gotResult
> _inlineCallbacks(r, g, deferred)
> --- <exception caught here> ---
> File "/usr/local/lib/python2.7/dist-
> packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
> result = result.throwExceptionIntoGenerator(g)
> File "/usr/local/lib/python2.7/dist-
> packages/twisted/python/failure.py", line 389, in
> throwExceptionIntoGenerator
> return g.throw(self.type, self.value, self.tb)
> File "/usr/local/lib/python2.7/dist-
> packages/buildbot/util/state.py", line 31, in getState
> **kwargs)
> File "/usr/local/lib/python2.7/dist-
> packages/twisted/python/threadpool.py", line 246, in inContext
> result = inContext.theWork()
> File "/usr/local/lib/python2.7/dist-
> packages/twisted/python/threadpool.py", line 262, in <lambda>
> inContext.theWork = lambda: context.call(ctx, func, *args,
> **kw)
> File "/usr/local/lib/python2.7/dist-
> packages/twisted/python/context.py", line 118, in callWithContext
> return self.currentContext().callWithContext(ctx, func,
> *args, **kw)
> File "/usr/local/lib/python2.7/dist-
> packages/twisted/python/context.py", line 81, in callWithContext
> return func(*args,**kw)
> File "/usr/local/lib/python2.7/dist-
> packages/buildbot/db/pool.py", line 175, in __thd
> rv = callable(arg, *args, **kwargs)
> File "/usr/local/lib/python2.7/dist-
> packages/buildbot/db/state.py", line 93, in thd
> return self.thdGetState(conn, objectid, name,
> default=default)
> File "/usr/local/lib/python2.7/dist-
> packages/buildbot/db/state.py", line 115, in thdGetState
> (name, objectid))
> exceptions.TypeError: JSON error loading state value 'lastRev'
> for 2
New description:
GitPoller stores a json value in the object_state MySQL table.
The type of value_json field is TEXT. TEXT is limited to 65,535 bytes.
When many branches with long names exist in the git repo, the json string
generated by GitPoller can be larger than 65,535 bytes. MySQL truncates
the string.
When GitPoller tries to read back the value, it does not find a well-
formed json string.
Exception :
{{{
2016-08-05 18:33:49+0000 [-] Got fatal Exception on DB
Traceback (most recent call last):
Failure: exceptions.TypeError: JSON error loading state value
'lastRev' for 2
2016-08-05 18:33:49+0000 [-] Starting factory
<twisted.web.client._HTTP11ClientFactory instance at 0x7f5bbd3f2cb0>
2016-08-05 18:33:49+0000 [-] while initializing GitPoller repository
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-
packages/twisted/internet/defer.py", line 434, in errback
self._startRunCallbacks(fail)
File "/usr/local/lib/python2.7/dist-
packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
self._runCallbacks()
File "/usr/local/lib/python2.7/dist-
packages/twisted/internet/defer.py", line 588, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/local/lib/python2.7/dist-
packages/twisted/internet/defer.py", line 1184, in gotResult
_inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
File "/usr/local/lib/python2.7/dist-
packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/local/lib/python2.7/dist-
packages/twisted/python/failure.py", line 389, in
throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/local/lib/python2.7/dist-
packages/buildbot/util/state.py", line 31, in getState
**kwargs)
File "/usr/local/lib/python2.7/dist-
packages/twisted/python/threadpool.py", line 246, in inContext
result = inContext.theWork()
File "/usr/local/lib/python2.7/dist-
packages/twisted/python/threadpool.py", line 262, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args,
**kw)
File "/usr/local/lib/python2.7/dist-
packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args,
**kw)
File "/usr/local/lib/python2.7/dist-
packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/usr/local/lib/python2.7/dist-
packages/buildbot/db/pool.py", line 175, in __thd
rv = callable(arg, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-
packages/buildbot/db/state.py", line 93, in thd
return self.thdGetState(conn, objectid, name, default=default)
File "/usr/local/lib/python2.7/dist-
packages/buildbot/db/state.py", line 115, in thdGetState
(name, objectid))
exceptions.TypeError: JSON error loading state value 'lastRev' for
2
}}}
--
--
Ticket URL: <http://trac.buildbot.net/ticket/3593#comment:2>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation
More information about the bugs
mailing list