[Buildbot-devel] 8-bit chars vs. sqlalchemy?

[Dustin J. Mitchell]

Buildbot does add the charset=utf-8:

https://github.com/buildbot/buildbot/blob/master/master/buildbot/db/enginestrategy.py#L129

In 0.8.x, more commonly the issue is feeding bytestrings to
sqlalchemy, rather than feeding unicode instances (which it could then
encode).  In fact, that's what the original error was, here.'

In nine, all access to the data API (and for the DB API) verifies that
it is receiving unicode instances and not bytestrings in all cases.
This leaves it up to the data source to determine the encoding.
That's not always easy -- for example, logfiles might contain
iso-8859-1 filenames and utf-8 output gcc, making an accurate decoding
very difficult.  Most data sources in nine (including logfiles) have
an encoding parameter that specifies the encoding they should expect
to find on their inputs.

It looks like this has not been done for
`master/buildbot/changes/mail.py`, not least because the file has
woefully inadequate test coverage.  That might make a nice task for
someone to get started hacking on nine..

Dustin

On Sat, Jan 4, 2014 at 12:27 AM, Ian Zimmerman <itz at buug.org> wrote:
> On Sun, 15 Dec 2013 11:01:03 -0500
> Charles Lepple <clepple at gmail.com> wrote:
>
> Charles> Maybe someone else can speak to the specifics of sqlalchemy and
> Charles> 8-bit characters. The part that caught my eye is that "Fu\xdf"
> Charles> (Fuß) is valid ISO 8859-1/15, but not valid UTF-8. Might be a
> Charles> red herring, though (I am not familiar with the code that
> Charles> printed the error message).
>
> This kind of stuff tends to happen when the sqlalchemy "engine" is
> opened with something like this:
>
> mysql://user:password@host/database
>
> instead of this:
>
> mysql://user:password@host/database?charset=utf-8
>
> I think the docs say the default is iso-8859-1, but in fact it is
> cp-1252 aka Western.
>
> --
> Please *no* private copies of mailing list or newsgroup messages.
>
> gpg public key: 2048R/984A8AE4
> fingerprint: 7953 ADA1 0E8E AB57 FB79  FFD2 360A 88B2 984A 8AE4
> Funny pic: http://bit.ly/ZNE2MX
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel