[Buildbot-devel] 8-bit chars vs. sqlalchemy?

[Charles Lepple]

On Nov 24, 2013, at 1:28 PM, Jeff Rizzo wrote:

> In troubleshooting why a particular commit didn't trigger a build, I 
> came across this in twistd.log:
> 
> sqlalchemy.exc.ProgrammingError: (ProgrammingError) You must not use 
> 8-bit bytestrings unless you use a text_factory that can interpret 8-bit 
> bytestrings (like text_factory = str). It is highly recommended that you 
> instead just switch your application to Unicode strings. u'INSERT INTO 
> changes (author, comments, is_dir, branch, revision, revlink, 
> when_timestamp, category, repository, codebase, project) VALUES (?, ?, 
> ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('bouyer', "XS_NOERROR | XS_RESET doens't 
> make sense, XS_RESET is what is wanted here.\nPointed out by Edgar 
> Fu\xdf\n", 0, None, '2013-11-24 18:02:08', '', 1385316128, 'kernel.', 
> 'anoncvs at anoncvs.netbsd.org:/cvsroot', '', 'NetBSD')

Just ran across this message - looks like Gmail thought it was spam.

Are you using the following ChangeSource with contrib/buildbot_cvs_mail.py?

http://docs.buildbot.net/current/manual/cfg-changesources.html#buildbot.changes.mail.CVSMaildirSource

> (full traceback is below)
> 
> Clearly, the commit log with the \xdf character is at issue - is there 
> anything simple I can do in my changesource to avoid this?  I would 
> prefer (obviously) not to strip out non-ascii characters, but I'm not 
> the most unicode-savvy...

Maybe someone else can speak to the specifics of sqlalchemy and 8-bit characters. The part that caught my eye is that "Fu\xdf" (Fuß) is valid ISO 8859-1/15, but not valid UTF-8. Might be a red herring, though (I am not familiar with the code that printed the error message).

> Full traceback:
> 
> 2013-11-24 18:02:28+0000 [-] while reading 
> '1385316139.12111_0.build.tastylime.net' from maildir '/home/buildbot/Mail':
>         Traceback (most recent call last):
>           File 
> "/usr/pkg/lib/python2.7/site-packages/twisted/internet/defer.py", line 
> 575, in _runCallbacks
>             current.result = callback(current.result, *args, **kw)
>           File 
> "/usr/pkg/lib/python2.7/site-packages/twisted/internet/defer.py", line 
> 1126, in gotResult
>             _inlineCallbacks(r, g, deferred)
>           File 
> "/usr/pkg/lib/python2.7/site-packages/twisted/internet/defer.py", line 
> 1068, in _inlineCallbacks
>             result = result.throwExceptionIntoGenerator(g)
>           File 
> "/usr/pkg/lib/python2.7/site-packages/twisted/python/failure.py", line 
> 389, in throwExceptionIntoGenerator
>             return g.throw(self.type, self.value, self.tb)
>         --- <exception caught here> ---
>           File 
> "/usr/pkg/lib/python2.7/site-packages/buildbot/util/maildir.py", line 
> 116, in poll
>             yield self.messageReceived(n)
>           File 
> "/usr/pkg/lib/python2.7/site-packages/twisted/python/threadpool.py", 
> line 191, in _worker
>             result = context.call(ctx, function, *args, **kwargs)
>           File 
> "/usr/pkg/lib/python2.7/site-packages/twisted/python/context.py", line 
> 118, in callWithContext
>             return self.currentContext().callWithContext(ctx, func, 
> *args, **kw)
>           File 
> "/usr/pkg/lib/python2.7/site-packages/twisted/python/context.py", line 
> 81, in callWithContext
>             return func(*args,**kw)
>           File 
> "/usr/pkg/lib/python2.7/site-packages/buildbot/db/pool.py", line 184, in 
> __thd
>             rv = callable(arg, *args, **kwargs)
>           File 
> "/usr/pkg/lib/python2.7/site-packages/buildbot/db/changes.py", line 78, 
> in thd
>             project=project))
>           File 
> "/usr/pkg/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 
> 1449, in execute
>             params)
>           File 
> "/usr/pkg/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 
> 1584, in _execute_clauseelement
>             compiled_sql, distilled_params
>           File 
> "/usr/pkg/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 
> 1698, in _execute_context
>             context)
>           File 
> "/usr/pkg/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 
> 1691, in _execute_context
>             context)
>           File 
> "/usr/pkg/lib/python2.7/site-packages/sqlalchemy/engine/default.py", 
> line 331, in do_execute
>             cursor.execute(statement, parameters)
>         sqlalchemy.exc.ProgrammingError: (ProgrammingError) You must 
> not use 8-bit bytestrings unless you use a text_factory that can 
> interpret 8-bit bytestrings (like text_factory = str). It is highly 
> recommended that you instead just switch your application to Unicode 
> strings. u'INSERT INTO changes (author, comments, is_dir, branch, 
> revision, revlink, when_timestamp, category, repository, codebase, 
> project) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('bouyer', 
> "XS_NOERROR | XS_RESET doens't make sense, XS_RESET is what is wanted 
> here.\nPointed out by Edgar Fu\xdf\n", 0, None, '2013-11-24 18:02:08', 
> '', 1385316128, 'kernel.', 'anoncvs at anoncvs.netbsd.org:/cvsroot', '', 
> 'NetBSD')

-- 
Charles Lepple
clepple at gmail