[Buildbot-commits] [Buildbot] #1056: handle encodings of change_files properly

[Buildbot]

#1056: handle encodings of change_files properly
-------------------------------+---------------------
Reporter:  dustin              |       Owner:  dustin
    Type:  enhancement         |      Status:  new
Priority:  major               |   Milestone:  0.8.4
 Version:  0.8.2               |  Resolution:
Keywords:  encoding, database  |
-------------------------------+---------------------

Comment (by dberger):

 I've run into what I believe is another instance of the same problem while
 working on upgrading from 0.7.11 to 0.8.3.

 We have a couple changes in the pickle that include files with non-ascii
 characters in their names (e.g. ...São Tomé & Príncipe...) - this was
 breaking the upgrade.  I wrote a simple bit of code to convert our pickle:

 import buildbot.changes
 import cPickle

 def recode( str ):
    try:
       s = str.decode("utf8")
       return str
    except:
       return str.decode("cp1252").encode("utf8")


 c = cPickle.load(open("changes.pck"))

 for change in c.changes:
    try:
       s = change.comments.decode("utf-8")
       change.comments = s
    except UnicodeDecodeError, e:
       try:
          s = change.comments.decode("cp1252")
          change.comments = s.encode("utf8")
       except UnicodeDecodeError, e:
          print change.comments

    change.files = map( recode, change.files )

 cPickle.dump( c, open("changes.pck.new", "w") )

 but the upgrade still failed, until I made the following change to v1.py

 $ p4 diff -du v1.py
 --- //.../tools/external/buildbot-0.8.3/buildbot/db/schema/v1.py
 2010-12-29 18:30:20.000000000 -0800
 +++ /.../tools/external/buildbot-0.8.3/buildbot/db/schema/v1.py 2010-12-29
 18:30:20.000000000 -0800
 @@ -291,7 +291,7 @@
                  return l
          for filename in flatten(change.files):
              cursor.execute(util.sql_insert(self.dbapi, 'change_files',
 ('changeid', 'filename')),
 -                          (change.number, filename))
 +                          (change.number, filename.decode("utf8")))
          for propname,propvalue in change.properties.properties.items():
              encoded_value = json.dumps(propvalue)
              cursor.execute(util.sql_insert(self.dbapi,
 'change_properties',

 but given this ticket I'm hesitant to move forward with the upgrade if
 there's a chance that a commit of a file with non-ascii characters in it's
 name will break the running buildbot instance.

-- 
Ticket URL: <http://trac.buildbot.net/ticket/1056#comment:3>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation