[Buildbot-commits] [Buildbot] #951: UTF-8 data in logs causes web rss feed exception

[Buildbot trac]

#951: UTF-8 data in logs causes web rss feed exception
--------------------+-----------------------
Reporter:  AHowell  |       Owner:  AHowell
    Type:  defect   |      Status:  reopened
Priority:  major    |   Milestone:  0.8.2
 Version:  0.8.1    |  Resolution:
Keywords:  web      |
--------------------+-----------------------
Changes (by paolog):

 * status:  closed => reopened
 * resolution:  fixed =>


Comment:

 Not fixed yet.

 I have this text in one of the logs: ""tar" non � riconosciuto come
 comando interno o esterno, un programma eseguibile o un file batch." -
 this was generated on an Windows 8 buildslave with Italian language
 settings.

 When I extract the feed with http://localhost:8080/atom?failures_only=true
 an a Debian 7 buildmaster (that' 0.8.6.) it returns:

 {{{
 web.Server Traceback (most recent call last):

 <type 'exceptions.UnicodeDecodeError'>: 'utf8' codec can't decode byte
 0x8a in position 10: invalid start byte
 /usr/lib/python2.7/dist-packages/twisted/web/server.py, line 132 in
 process
 130    try:
 131      resrc = self.site.getResourceFor(self)
 132      self.render(resrc)
 133    except:
 /usr/lib/python2.7/dist-packages/twisted/web/server.py, line 167 in render
 165    """
 166    try:
 167      body = resrc.render(self)
 168    except UnsupportedMethod, e:
 /usr/lib/python2.7/dist-packages/buildbot/status/web/feeds.py, line 55 in
 render
 53
 54  def render(self, request):
 55    data = self.content(request)
 56    request.setHeader("content-type", self.contentType)
 /usr/lib/python2.7/dist-packages/buildbot/status/web/feeds.py, line 225 in
 content
 223            for line in logdata.split('\n')[-30:]:
 224              # unilist.append(unicode(line,'utf-8','replace'))
 225              unilist.append(unicode(line,'utf-8'))
 226            log_lines.extend(unilist)

 <type 'exceptions.UnicodeDecodeError'>: 'utf8' codec can't decode byte
 0x8a in position 10: invalid start byte
 }}}

 The following fix works: in /usr/lib/python2.7/dist-
 packages/buildbot/status/web/feeds.py at line 224 change

 {{{unilist.append(unicode(line,'utf-8'))}}}

 to

 {{{unilist.append(unicode(line,'utf-8','replace'))}}}

 see
 http://docs.python.org/2/library/functions.html?highlight=unicode#unicode

-- 
Ticket URL: <http://trac.buildbot.net/ticket/951#comment:5>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation