[Buildbot-commits] [SPAM] [Buildbot] #820: Change imports fail on 8-bit comments

[Buildbot]

#820: Change imports fail on 8-bit comments
-------------------+--------------------------------------------------------
Reporter:  dustin  |       Owner:       
    Type:  defect  |      Status:  new  
Priority:  major   |   Milestone:  0.8.0
 Version:  0.8.0   |    Keywords:       
-------------------+--------------------------------------------------------
 We need to handle this with something more graceful than a traceback!

 {{{
 [builds at lzeno buildbot]$ buildbot upgrade-master .
 /opt/local/buildbot/public_html/robots.txt has old/modified contents
  writing new contents to /opt/local/buildbot/public_html/robots.txt.new
 /opt/local/buildbot/public_html/bg_gradient.jpg has old/modified contents
  writing new contents to
 /opt/local/buildbot/public_html/bg_gradient.jpg.new
 migrating changes.pck to database
  (12255 Change objects)
 Traceback (most recent call last):
  File "/tools/linux-ia32/python/python/2.5.4/bin/buildbot", line 5, in
 <module>
    pkg_resources.run_script('buildbot==0.8.0rc2', 'buildbot')
  File "/tools/linux-ia32/python/python/2.5.4/lib/python2.5/site-
 packages/setuptools-0.6c11-py2.5.egg/pkg_resources.py",
 line 489, in run_script
  File "/tools/linux-ia32/python/python/2.5.4/lib/python2.5/site-
 packages/setuptools-0.6c11-py2.5.egg/pkg_resources.py",
 line 1207, in run_script
  File "/tools/linux-ia32/python/python/2.5.4/lib/python2.5/site-
 packages/buildbot-0.8.0rc2-py2.5.egg/EGG-INFO/scripts/buildbot",
 line 4, in <module>
    runner.run()
  File "/tools/linux-ia32/python/python/2.5.4/lib/python2.5/site-
 packages/buildbot-0.8.0rc2-py2.5.egg/buildbot/scripts/runner.py",
 line 1193, in run
    upgradeMaster(so)
  File "/tools/linux-ia32/python/python/2.5.4/lib/python2.5/site-
 packages/buildbot-0.8.0rc2-py2.5.egg/buildbot/scripts/runner.py",
 line 453, in upgradeMaster
    sm.upgrade()
  File "/tools/linux-ia32/python/python/2.5.4/lib/python2.5/site-
 packages/buildbot-0.8.0rc2-py2.5.egg/buildbot/db/schema/manager.py",
 line 56, in upgrade
    upg.upgrade()
  File "/tools/linux-ia32/python/python/2.5.4/lib/python2.5/site-
 packages/buildbot-0.8.0rc2-py2.5.egg/buildbot/db/schema/v1.py",
 line 245, in upgrade
    self.migrate_changes()
  File "/tools/linux-ia32/python/python/2.5.4/lib/python2.5/site-
 packages/buildbot-0.8.0rc2-py2.5.egg/buildbot/db/schema/v1.py",
 line 317, in migrate_changes
    self._addChangeToDatabase(c, cursor)
  File "/tools/linux-ia32/python/python/2.5.4/lib/python2.5/site-
 packages/buildbot-0.8.0rc2-py2.5.egg/buildbot/db/schema/v1.py",
 line 271, in _addChangeToDatabase
    cursor.execute(q, values)
 pysqlite2.dbapi2.ProgrammingError: You must not use 8-bit bytestrings
 unless you use a text_factory that can interpret 8-bit bytestrings
 (like text_factory = str). I t is highly recommended that you instead
 just switch your application to Unicode strings.
 }}}
 ----
 More info.
 Looks like at least one of the comments for the perforce checkin's had
 unicode characters.
 I hacked this as follows

 {{{
    def _addChangeToDatabase(self, change, cursor):
        # strip None from any of these values, just in case
        def remove_none(x):
            if x is None: return ""
            return x

        # Hack - de-unicode the comments
        try:
            values = tuple(remove_none(x) for x in
                           (change.number, change.who,
                            change.comments, change.isdir,
                            change.branch, change.revision, change.revlink,
                            change.when, change.category))
            q = util.sql_insert(self.dbapi, 'changes',
                                """changeid author comments is_dir branch
 revision
                                revlink when_timestamp category""".split())

 #            print "Values :%s"%(" ".join([str(a) for a in values]))
            cursor.execute(q, values)
        except pysqlite2.dbapi2.ProgrammingError:
            values = tuple(remove_none(x) for x in
                           (change.number, change.who,
                            "dropped comments due to unicode",
 change.isdir,
                            change.branch, change.revision, change.revlink,
                            change.when, change.category))
            q = util.sql_insert(self.dbapi, 'changes',
                                """changeid author comments is_dir branch
 revision
                                revlink when_timestamp category""".split())
            cursor.execute(q, values)
 }}}
 Note that after upgrade, buildbot seems to pound on the CPU more.

-- 
Ticket URL: <http://buildbot.net/trac/ticket/820>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation