From warner at users.sourceforge.net  Mon Oct  2 00:13:13 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:13 +0000
Subject: [Buildbot-commits] buildbot ChangeLog.svnpoller,NONE,1.1
Message-ID: <E1GUBQo-0000JT-Lw@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17108

Added Files:
	ChangeLog.svnpoller 
Log Message:
[project @ svnpoller: add new ChangeSource, interim checkin]

Original author: warner at lothar.com
Date: 2006-10-01 17:18:18

--- NEW FILE: ChangeLog.svnpoller ---
Sun Oct  1 10:12:44 2006  Brian Warner  <warner at lothar.com>

	* buildbot/changes/svnpoller.py (SvnSource): added Niklaus Giger's
	Suvbersion repository polling ChangeSource. I've hacked it up
	considerably: any bugs are entirely my own fault. Thank you
	Niklaus!
	* buildbot/test/test_svnpoller.py: tests for it
	* docs/buildbot.texinfo (SvnSource): document it




From warner at users.sourceforge.net  Mon Oct  2 00:13:13 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:13 +0000
Subject: [Buildbot-commits] buildbot/buildbot/changes svnpoller.py, NONE, 1.1
Message-ID: <E1GUBQo-0005Az-M1@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/changes
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17108/buildbot/changes

Added Files:
	svnpoller.py 
Log Message:
[project @ svnpoller: add new ChangeSource, interim checkin]

Original author: warner at lothar.com
Date: 2006-10-01 17:18:18

--- NEW FILE: svnpoller.py ---
# -*- test-case-name: buildbot.test.test_svnpoller -*-

# Based on the work of Dave Peticolas for the P4poll
# Changed to svn (using xml.dom.minidom) by Niklaus Giger

import time

from twisted.python import log, failure
from twisted.internet import defer, reactor
from twisted.internet.utils import getProcessOutput
from twisted.internet.task import LoopingCall

from buildbot import util
from buildbot.changes import base
from buildbot.changes.changes import Change

import xml.dom.minidom

def _assert(condition, msg):
    if condition:
        return True
    raise AssertionError(msg)

def dbgMsg(myString):
    log.msg(myString)
    return 1

# these split_file_* functions are available for use as values to the
# split_file= argument.
def split_file_alwaystrunk(path):
    return (None, path)

def split_file_branches(path):
    pieces = path.split('/')
    if pieces[0] == 'trunk':
        return (None, '/'.join(pieces[1:]))
    elif pieces[0] == 'branches':
        return (pieces[1], '/'.join(pieces[2:]))
    else:
        return None


class SvnSource(base.ChangeSource, util.ComparableMixin):
    """This source will poll a perforce repository for changes and submit
    them to the change master."""

    compare_attrs = ["svnuser", "svnpasswd", "svnurl",
                     "pollinterval", "histmax"]

    parent = None # filled in when we're added
    last_change = None
    loop = None
    working = False

    def __init__(self, svnurl, split_file=None,
                 svnuser=None, svnpasswd=None,
                 svnbin='svn',
                 pollinterval=10*60, histmax=100):
        """
        @type  svnurl: string
        @param svnurl: the SVN URL that describes the repository and
                       subdirectory to watch. If this ChangeSource should
                       only pay attention to a single branch, this should
                       point at the repository for that branch, like
                       svn://svn.twistedmatrix.com/svn/Twisted/trunk . If it
                       should follow multiple branches, point it at the
                       repository directory that contains all the branches
                       like svn://svn.twistedmatrix.com/svn/Twisted and also
                       provide a branch-determining function.

                       Each file in the repository has a SVN URL in the form
                       (SVNURL)/(BRANCH)/(FILEPATH), where (BRANCH) could be
                       empty or not, depending upon your branch-determining
                       function. Only files that start with (SVNURL)/(BRANCH)
                       will be monitored. The Change objects that are sent to
                       the Schedulers will see (FILEPATH) for each modified
                       file.

        @type  split_file: callable or None
        @param split_file: a function that is called with a string of the
                           form (BRANCH)/(FILEPATH) and should return a tuple
                           (BRANCH, FILEPATH). This function should match
                           your repository's branch-naming policy. Each
                           changed file has a fully-qualified URL that can be
                           split into a prefix (which equals the value of the
                           'svnurl' argument) and a suffix; it is this suffix
                           which is passed to the split_file function.

                           If the function returns None, the file is ignored.
                           Use this to indicate that the file is not a part
                           of this project.
                           
                           For example, if your repository puts the trunk in
                           trunk/... and branches are in places like
                           branches/1.5/..., your split_file function could
                           look like the following (this function is
                           available as svnpoller.split_file_branches)::

                            pieces = path.split('/')
                            if pieces[0] == 'trunk':
                                return (None, '/'.join(pieces[1:]))
                            elif pieces[0] == 'branches':
                                return (pieces[1], '/'.join(pieces[2:]))
                            else:
                                return None

                           If instead your repository layout puts the trunk
                           for ProjectA in trunk/ProjectA/... and the 1.5
                           branch in branches/1.5/ProjectA/..., your
                           split_file function could look like::

                            pieces = path.split('/')
                            if pieces[0] == 'trunk':
                                branch = None
                                pieces.pop(0) # remove 'trunk'
                            elif pieces[0] == 'branches':
                                pieces.pop(0) # remove 'branches'
                                branch = pieces.pop(0) # grab branch name
                            else:
                                return None # something weird
                            projectname = pieces.pop(0)
                            if projectname != 'ProjectA':
                                return None # wrong project
                            return (branch, '/'.join(pieces))

                           The default of split_file= is None, which
                           indicates that no splitting should be done. This
                           is equivalent to the following function::

                            return (None, path)

                           If you wish, you can override the split_file
                           method with the same sort of function instead of
                           passing in a split_file= argument.


        @type  svnuser:      string
        @param svnuser:      If set, the --username option will be added to
                             the 'svn log' command. You may need this to get
                             access to a private repository.
        @type  svnpasswd:    string
        @param svnpasswd:    If set, the --password option will be added.

        @type  svnbin:       string
        @param svnbin:       path to svn binary, defaults to just 'svn'. Use
                             this if your subversion command lives in an
                             unusual location.

        @type  pollinterval: int
        @param pollinterval: interval in seconds between polls. The default
                             is 600 seconds (10 minutes). Smaller values
                             decrease the latency between the time a change
                             is recorded and the time the buildbot notices
                             it, but it also increases the system load.

        @type  histmax:      int
        @param histmax:      maximum number of changes to look back through.
                             The default is 100. Smaller values decrease
                             system load, but if more than histmax changes
                             are recorded between polls, the extra ones will
                             be silently lost.
        """

        if svnurl.endswith("/"):
            svnurl = svnurl[:-1] # strip the trailing slash
        self.svnurl = svnurl
        self.split_file_function = split_file or split_file_alwaystrunk
        self.svnuser = svnuser
        self.svnpasswd = svnpasswd

        self.svnbin = svnbin
        self.pollinterval = pollinterval
        self.histmax = histmax
        self._root = None
        self.loop = LoopingCall(self.checksvn)

    def split_file(self, path):
        # use getattr() to avoid turning this function into a bound method,
        # which would require it to have an extra 'self' argument
        f = getattr(self, "split_file_function")
        return f(path)

    def startService(self):
        base.ChangeSource.startService(self)
        # Don't start the loop just yet because the reactor isn't running.
        # Give it a chance to go and install our SIGCHLD handler before
        # spawning processes.
        reactor.callLater(0, self.loop.start, self.pollinterval)

    def stopService(self):
        self.loop.stop()
        return base.ChangeSource.stopService(self)

    def describe(self):
        return "svnsource %s branch %s" % ( self.svnurl, self.branch)

    def checksvn(self):
        # Our return value is only used for unit testing.

        # we need to figure out the repository root, so we can figure out
        # repository-relative pathnames later. Each SVNURL is in the form
        # (ROOT)/(PROJECT)/(BRANCH)/(FILEPATH), where (ROOT) is something
        # like svn://svn.twistedmatrix.com/svn/Twisted (i.e. there is a
        # physical repository at /svn/Twisted on that host), (PROJECT) is
        # something like Projects/Twisted (i.e. within the repository's
        # internal namespace, everything under Projects/Twisted/ has
        # something to do with Twisted, but these directory names do not
        # actually appear on the repository host), (BRANCH) is something like
        # "trunk" or "branches/2.0.x", and (FILEPATH) is a tree-relative
        # filename like "twisted/internet/defer.py".

        # our self.svnurl attribute contains (ROOT)/(PROJECT) combined
        # together in a way that we can't separate without svn's help. If the
        # user is not using the split_file= argument, then self.svnurl might
        # be (ROOT)/(PROJECT)/(BRANCH) . In any case, the filenames we will
        # get back from 'svn log' will be of the form
        # (PROJECT)/(BRANCH)/(FILEPATH), but we want to be able to remove
        # that (PROJECT) prefix from them. To do this without requiring the
        # user to tell us how svnurl is split into ROOT and PROJECT, we do an
        # 'svn info --xml' command at startup. This command will include a
        # <root> element that tells us ROOT. We then strip this prefix from
        # self.svnurl to determine PROJECT, and then later we strip the
        # PROJECT prefix from the filenames reported by 'svn log --xml' to
        # get a (BRANCH)/(FILEPATH) that can be passed to split_file() to
        # turn into separate BRANCH and FILEPATH values.

        # whew.

        if not self._prefix:
            # this sets self._prefix when it finishes
            d = self._determine_prefix()
        else:
            d = defer.succeed(None)

        if self.working:
            dbgMsg("Skipping checksvn because last one has not finished")
            return d

        self.working = True
        d.addCallback(lambda res: self._get_logs())
        d.addCallback(self._parse_logs)
        d.addCallback(self._get_new_logentries)
        d.addCallback(self._create_changes)
        d.addCallback(self._submit_changes)
        d.addBoth(self._finished)
        return d

    def _determine_prefix(self):
        args = ["info", "--xml", "--non-interactive", self.svnurl]
        if self.svnuser:
            args.extend(["--username=%s" % self.svnuser])
        if self.svnpasswd:
            args.extend(["--password=%s" % self.svnpasswd])
        d = getProcessOutput(self.svnbin, args, {})
        d.addCallback(self._determine_prefix_2)
        return d

    def _determine_prefix_2(self, output):
	try:
	    doc = xml.dom.minidom.parseString(output)
	except xml.parsers.expat.ExpatError:
	    dbgMsg("_process_changes: ExpatError in %s" % output)
            log.msg("SvnSource._determine_prefix_2: ExpatError in '%s'"
                    % output)
	    raise
        rootnodes = doc.getElementsByTagName("root")
        if not rootnodes:
            # this happens if the URL we gave was already the root. In this
            # case, our prefix is empty.
            self._prefix = ""
            return self._prefix
        rootnode = rootnodes[0]
        root = "".join([c.data for c in rootnode.childNodes])
        # root will be a unicode string
        _assert(self.svnurl.startswith(root),
                "svnurl='%s' doesn't start with <root>='%s'" %
                (self.svnurl, root))
        self._prefix = self.svnurl[len(root):]
        if self._prefix.startswith("/"):
            self._prefix = self._prefix[1:]
        log.msg("SvnSource: svnurl=%s, root=%s, so prefix=%s" %
                (self.svnurl, root, self._prefix))
        return self._prefix

    def _get_logs(self):
        args = []
        args.extend(["log", "--xml", "--verbose", "--non-interactive"])
        if self.svnuser:
            args.extend(["--username=%s" % self.svnuser])
        if self.svnpasswd:
            args.extend(["--password=%s" % self.svnpasswd])
        args.extend(["--limit=%d" % (self.histmax), self.svnurl])
        env = {}
        res = getProcessOutput(self.svnbin, args, env)
	log.msg("_get_changes args %s end %s returns %s" % (args, env, res))
	return res

    def _parse_logs(self, output):
        # parse the XML output, return a list of <logentry> nodes
	try:
	    doc = xml.dom.minidom.parseString(output)
	except xml.parsers.expat.ExpatError:
	    dbgMsg("_process_changes: ExpatError in %s" % output)
            log.msg("SvnSource._parse_changes: ExpatError in '%s'" % output)
            raise
        logentries = doc.getElementsByTagName("logentry")
        return logentries

    def _filter_new_logentries(self, logentries, last_change):
        # given a list of logentries, return a tuple of (new_last_change,
        # new_logentries), where new_logentries contains only the ones after
        # last_change
        if not logentries:
            # no entries, so last_change must stay at None
            return (None, [])

	mostRecent = int(logentries[0].getAttribute("revision"))

        if last_change is None:
            # if this is the first time we've been run, ignore any changes
            # that occurred before now. This prevents a build at every
            # startup.
	    log.msg('svnPoller: starting at change %s' % mostRecent)
	    self.last_change = mostRecent
	    return (mostRecent, [])

	if last_change == mostRecent:
            # an unmodified repository will hit this case
	    log.msg('svnPoller: _process_changes last %s mostRecent %s' % (
		      last_change, mostRecent))
            return (mostRecent, [])

        new_logentries = []
	for el in logentries:
            if last_change == int(el.getAttribute("revision")):
		break
            new_logentries.append(el)
        new_logentries.reverse() # return oldest first
        return (mostRecent, new_logentries)

    def _get_text(self, element, tag_name):
        child_nodes = element.getElementsByTagName(tag_name)[0].childNodes
        text = "".join([t.data for t in child_nodes])
        return text

    def _transform_path(self, path):
        _assert(path.startswith(self._prefix),
                "filepath '%s' should start with prefix '%s'" %
                (path, self._prefix))
        relative_path = path[len(self._prefix):]
        if relative_path.startswith("/"):
            relative_path = relative_path[1:]
        branch, final_path = self.split_file(relative_path)
        return branch, final_path

    def _get_new_logentries(self, logentries):
        last_change = self.last_change
        (new_last_change,
         new_logentries) = self._filter_new_logentries(logentries,
                                                       self.last_change)
        self.last_change = new_last_change
	log.msg('svnPoller: _process_changes %s .. %s' %
                (last_change, new_last_change))
        return new_logentries

    def _create_changes(self, new_logentries):
        changes = []

        for el in new_logentries:
	    branch_files = [] # get oldest change first
	    revision = int(el.getAttribute("revision"))
	    dbgMsg("Adding change revision %s" % (revision,))
	    author   = self._get_text(el, "author")
	    comments = self._get_text(el, "msg")
	    when     = self._get_text(el, "date")
	    when     = time.mktime(time.strptime("%.19s" % when,
                                                 "%Y-%m-%dT%H:%M:%S"))
            branches = {}
	    pathlist = el.getElementsByTagName("paths")[0]
	    for p in pathlist.getElementsByTagName("path"):
		path = "".join([t.data for t in p.childNodes])
                if path.startswith("/"):
                    path = path[1:]
                branch, filename = self._transform_path(path)
                if not branch in branches:
                    branches[branch] = []
                branches[branch].append(filename)

            for branch in branches:
                c = Change(who=author,
                           files=branches[branch],
                           comments=comments,
                           revision=revision,
                           when=when,
                           branch=branch)
                changes.append(c)

        return changes

    def _submit_changes(self, changes):
        for c in changes:
	    self.parent.addChange(c)

    def _finished(self, res):
        dbgMsg('_finished : %s' % res)
        assert self.working
        self.working = False

        # Again, the return value is only for unit testing.
        # If there's a failure, log it so it isn't lost.
        if isinstance(res, failure.Failure):
            dbgMsg('checksvn failed: %s' % res)
        return res




From warner at users.sourceforge.net  Mon Oct  2 00:13:14 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:14 +0000
Subject: [Buildbot-commits] buildbot/buildbot/test test_svnpoller.py, NONE,
	1.1
Message-ID: <E1GUBQo-0005B7-Qa@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/test
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17108/buildbot/test

Added Files:
	test_svnpoller.py 
Log Message:
[project @ svnpoller: add new ChangeSource, interim checkin]

Original author: warner at lothar.com
Date: 2006-10-01 17:18:18

--- NEW FILE: test_svnpoller.py ---
# Here are the tests for the SvnSource

import sys
import time

from twisted.python import log, failure
from twisted.internet import defer
from twisted.trial import unittest

from buildbot.twcompat import maybeWait
from buildbot.changes.changes import Change
from buildbot.changes.svnpoller import SvnSource, split_file_branches

# was changes 1012 in xenomai.org
svn_change_1 = """<?xml version="1.0" encoding="utf-8"?>
<log>
<logentry
   revision="101">
<author>rpm</author>
<date>2006-05-17T14:58:28.494960Z</date>
<paths>
<path
   action="M">/branch/ksrc/arch/i386/hal.c</path>
</paths>
<msg>Remove unused variable</msg>
</logentry>
</log>
"""

svn_change_2 = """<?xml version="1.0" encoding="utf-8"?>
<log>
<logentry
   revision="102">
<author>rpm</author>
<date>2006-05-15T12:54:08.891420Z</date>
<paths>
<path
   action="M">/trunk/ChangeLog</path>
<path
   action="D">/trunk/ksrc/first_file</path>
<path
   action="D">/trunk/ksrc/second_file</path>
</paths>
<msg>Initial Adeos support</msg>
</logentry>
</log>
"""

svn_change_3 = """<?xml version="1.0" encoding="utf-8"?>
<log>
<logentry
   revision="102">
<author>rpm</author>
<date>2006-05-15T12:54:08.891420Z</date>
<paths>
<path
   action="M">/trunk/ChangeLog</path>
<path
   action="D">/trunk/ksrc/first_file</path>
<path
   action="D">/trunk/ksrc/second_file</path>
</paths>
<msg>Upgrade Adeos support</msg>
</logentry>
</log>
"""

def dbgMsg(myString):
    log.msg(myString)
    return 1

class MockSvnSource(SvnSource):
    """Test SvnSource which doesn't actually invoke svn."""
    invocation = 0

    def __init__(self, svnchanges, *args, **kwargs):
        SvnSource.__init__(self, None, *args, **kwargs)
        self.svnchanges = svnchanges

    def _get_changes(self):
        assert self.working
        result = self.svnchanges[self.invocation]
        self.invocation += 1
        #log.msg("MockSvnSource._get_changes %s result %s " % (self.invocation-1, result))
        dbgMsg("MockSvnSource._get_changes %s " % (self.invocation-1))
        return defer.succeed(result)

    def _get_describe(self, dummy, num):
        assert self.working
        dbgMsg("MockSvnSource._get_describe %s " % num)
        return defer.succeed(self.svnchanges[num])

class TestSvnPoller(unittest.TestCase):
    def setUp(self):
        self.changes = []
        self.addChange = self.changes.append

    def failUnlessIn(self, substr, string):
        # this is for compatibility with python2.2
        if isinstance(string, str):
            self.failUnless(string.find(substr) != -1)
        else:
            self.assertIn(substr, string)

    def testCheck(self):
        """successful checks"""
        self.t = MockSvnSource(svnchanges=[ svn_change_1, svn_change_2, svn_change_3],
                              svnuser=None,
                              svnurl='/trunk/',)
        self.t.parent = self

        # The first time, it just learns the change to start at
        self.assert_(self.t.last_change is None)
        self.assert_(not self.t.working)
	dbgMsg("zzz")
        return maybeWait(self.t.checksvn().addCallback(self._testCheck2))

    def _testCheck2(self, res):
	dbgMsg("zzz1")
        self.assertEquals(self.changes, [])
	dbgMsg("zzz2 %s %s" % (self.t.last_change, self.changes))
        self.assertEquals(self.t.last_change, '101')
	dbgMsg("zzz3")

        # Subsequent times, it returns Change objects for new changes.
        return self.t.checksvn().addCallback(self._testCheck3)

    def _testCheck3(self, res):
        # They're supposed to go oldest to newest, so this one must be first.
	tstChange1 = Change(who='rpm',
                   files=['/trunk/ChangeLog',
		          '/trunk/ksrc/first_file',
			  '/trunk/ksrc/second_file'],
                   comments="Initial Adeos support",
                   revision='2',
                   when=self.makeTime("2006/05/15 12:54:08"),
                   branch='trunk').asText()
	dbgMsg("tstChange" + tstChange1)
	dbgMsg("changes[0]" + self.changes[0].asText())

	self.assertEquals(self.changes[0].asText(), tstChange1)
        # Subsequent times, it returns Change objects for new changes.
        return self.t.checksvn().addCallback(self._testCheck4)

    def _testCheck4(self, res):
	dbgMsg("zzz5 %s " % len(self.changes))
        self.assertEquals(len(self.changes), 1)
	dbgMsg("zzz6 %s %s" % (self.t.last_change, self.changes))
        self.assertEquals(self.t.last_change, '102')
	dbgMsg("zzz7")
        self.assert_(not self.t.working)
	tstChange2 = Change(who='rpm',
                   files=['/trunk/ChangeLog',
		          '/trunk/ksrc/first_file',
			  '/trunk/ksrc/second_file'],
                   comments="Initial Adeos support",
                   revision='2',
                   when=self.makeTime("2006/05/15 12:54:08"),
                   branch='trunk').asText()
	dbgMsg("changes[0]" + self.changes[0].asText())
	dbgMsg("tstChange2" + tstChange2)
        self.assertEquals(self.changes[0].asText(), tstChange2)
	dbgMsg(7777)

    def makeTime(self, timestring):
        datefmt = '%Y/%m/%d %H:%M:%S'
        when = time.mktime(time.strptime(timestring, datefmt))
        return when

    def testFailedChanges(self):
        """'svn changes' failure is properly reported"""
        self.t = MockSvnSource(svnchanges=['Subversion client error:\n...'],
                               svnuser=None,
                               svnurl="/trunk")
        self.t.parent = self
        d = self.t.checksvn()
        d.addBoth(self._testFailedChanges2)
        return maybeWait(d)

    def _testFailedChanges2(self, f):
        self.assert_(isinstance(f, failure.Failure))
        self.failUnlessIn('Subversion client error', str(f))
        self.assert_(not self.t.working)

    def testFailedDescribe(self):
        """'svn describe' failure is properly reported"""
        self.t = MockSvnSource(svnchanges=[
	                       svn_change_1,
			       'Subversion client error:\n...',
			       svn_change_2,],
                               svnuser=None)
        self.t.parent = self
        d = self.t.checksvn()
	dbgMsg("xxx")
        d.addCallback(self._testFailedDescribe2)
        return maybeWait(d)

    def _testFailedDescribe2(self, res):
        # first time finds nothing; check again.
	dbgMsg("yy")
        res = self.t.checksvn().addBoth(self._testFailedDescribe3)
        return res

    def _testFailedDescribe3(self, f):
	dbgMsg("yy1 %s" % f)
        self.assert_(isinstance(f, failure.Failure))
	dbgMsg("yy2")
        self.failUnlessIn('Subversion client error', str(f))
	dbgMsg("yy3")
        self.assert_(not self.t.working)
	dbgMsg("yy4")
        self.assertEquals(self.t.last_change, '101')
	dbgMsg("yy5")

    def testAlreadyWorking(self):
        """don't launch a new poll while old is still going"""
        self.t = SvnSource()
        self.t.working = True
        self.assert_(self.t.last_change is None)
        d = self.t.checksvn()
        d.addCallback(self._testAlreadyWorking2)

    def _testAlreadyWorking2(self, res):
        self.assert_(self.t.last_change is None)

# this is the output of "svn info --xml
# svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk"
prefix_output = """\
<?xml version="1.0"?>
<info>
<entry
   kind="dir"
   path="trunk"
   revision="18354">
<url>svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk</url>
<repository>
<root>svn+ssh://svn.twistedmatrix.com/svn/Twisted</root>
<uuid>bbbe8e31-12d6-0310-92fd-ac37d47ddeeb</uuid>
</repository>
<commit
   revision="18352">
<author>jml</author>
<date>2006-10-01T02:37:34.063255Z</date>
</commit>
</entry>
</info>
"""

# and this is "svn info --xml svn://svn.twistedmatrix.com/svn/Twisted". I
# think this is kind of a degenerate case.. it might even be a form of error.
prefix_output_2 = """\
<?xml version="1.0"?>
<info>
</info>
"""

# this is the svn info output for a local repository, svn info --xml
# file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository
prefix_output_3 = """\
<?xml version="1.0"?>
<info>
<entry
   kind="dir"
   path="SVN-Repository"
   revision="3">
<url>file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository</url>
<repository>
<root>file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository</root>
<uuid>c0f47ff4-ba1e-0410-96b5-d44cc5c79e7f</uuid>
</repository>
<commit
   revision="3">
<author>warner</author>
<date>2006-10-01T07:37:04.182499Z</date>
</commit>
</entry>
</info>
"""

# % svn info --xml file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample/trunk

prefix_output_4 = """\
<?xml version="1.0"?>
<info>
<entry
   kind="dir"
   path="trunk"
   revision="3">
<url>file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample/trunk</url>
<repository>
<root>file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository</root>
<uuid>c0f47ff4-ba1e-0410-96b5-d44cc5c79e7f</uuid>
</repository>
<commit
   revision="1">
<author>warner</author>
<date>2006-10-01T07:37:02.286440Z</date>
</commit>
</entry>
</info>
"""



class ComputePrefix(unittest.TestCase):
    def test1(self):
        base = "svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk"
        s = SvnSource(base + "/")
        self.failUnlessEqual(s.svnurl, base) # certify slash-stripping
        prefix = s._determine_prefix_2(prefix_output)
        self.failUnlessEqual(prefix, "trunk")
        self.failUnlessEqual(s._prefix, prefix)

    def test2(self):
        base = "svn+ssh://svn.twistedmatrix.com/svn/Twisted"
        s = SvnSource(base)
        self.failUnlessEqual(s.svnurl, base)
        prefix = s._determine_prefix_2(prefix_output_2)
        self.failUnlessEqual(prefix, "")

    def test3(self):
        base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository"
        s = SvnSource(base)
        self.failUnlessEqual(s.svnurl, base)
        prefix = s._determine_prefix_2(prefix_output_3)
        self.failUnlessEqual(prefix, "")

    def test4(self):
        base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample/trunk"
        s = SvnSource(base)
        self.failUnlessEqual(s.svnurl, base)
        prefix = s._determine_prefix_2(prefix_output_4)
        self.failUnlessEqual(prefix, "sample/trunk")

# output from svn log on .../SVN-Repository/sample
# (so it includes trunk and branches)
changes_output_1 = """\
<?xml version="1.0"?>
<log>
<logentry
   revision="3">
<author>warner</author>
<date>2006-10-01T07:37:04.182499Z</date>
<paths>
<path
   action="M">/sample/branch/main.c</path>
</paths>
<msg>commit_on_branch</msg>
</logentry>
<logentry
   revision="2">
<author>warner</author>
<date>2006-10-01T07:37:03.175326Z</date>
<paths>
<path
   copyfrom-path="/sample/trunk"
   copyfrom-rev="1"
   action="A">/sample/branch</path>
</paths>
<msg>make_branch</msg>
</logentry>
</log>
"""

class ComputeChanges(unittest.TestCase):
    def test1(self):
        base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample"
        s = SvnSource(base)
        s._prefix = "sample"
        doc = s._parse_logs(changes_output_1)

        newlast, logentries = s._filter_new_logentries(doc, 3)
        self.failUnlessEqual(newlast, 3)
        self.failUnlessEqual(len(logentries), 0)

        newlast, logentries = s._filter_new_logentries(doc, 2)
        self.failUnlessEqual(newlast, 3)
        self.failUnlessEqual(len(logentries), 1)

        newlast, logentries = s._filter_new_logentries(doc, 0)
        self.failUnlessEqual(newlast, 3)
        self.failUnlessEqual(len(logentries), 2)

    def split_file(self, path):
        pieces = path.split("/")
        if pieces[0] == "branch":
            return "branch", "/".join(pieces[1:])
        if pieces[0] == "trunk":
            return None, "/".join(pieces[1:])
        raise RuntimeError("there shouldn't be any files like %s" % path)

    def testChanges(self):
        base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample"
        s = SvnSource(base, split_file=self.split_file)
        s._prefix = "sample"
        doc = s._parse_logs(changes_output_1)
        newlast, logentries = s._filter_new_logentries(doc, 0)
        changes = s._create_changes(logentries)
        self.failUnlessEqual(len(changes), 2)
        self.failUnlessEqual(changes[0].branch, "branch")
        self.failUnlessEqual(changes[1].branch, "branch")
        self.failUnlessEqual(changes[1].files, ["main.c"])




From warner at users.sourceforge.net  Mon Oct  2 00:13:14 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:14 +0000
Subject: [Buildbot-commits] buildbot/docs buildbot.texinfo,1.81,1.82
Message-ID: <E1GUBQo-0005BC-TA@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/docs
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17108/docs

Modified Files:
	buildbot.texinfo 
Log Message:
[project @ svnpoller: add new ChangeSource, interim checkin]

Original author: warner at lothar.com
Date: 2006-10-01 17:18:18

Index: buildbot.texinfo
===================================================================
RCS file: /cvsroot/buildbot/buildbot/docs/buildbot.texinfo,v
retrieving revision 1.81
retrieving revision 1.82
diff -u -d -r1.81 -r1.82
--- buildbot.texinfo	30 Sep 2006 21:12:08 -0000	1.81
+++ buildbot.texinfo	2 Oct 2006 00:13:12 -0000	1.82
@@ -153,6 +153,7 @@
 * PBChangeSource::              
 * P4Source::                    
 * BonsaiPoller::                
+* SvnSource::                   
 
 Build Process
 
@@ -2525,6 +2526,7 @@
 a suitable @code{MaildirSource}.
 
 
+* SvnSource::                    
 @menu
 * Choosing ChangeSources::      
 * CVSToys - PBService::         
@@ -2533,6 +2535,7 @@
 * PBChangeSource::              
 * P4Source::                    
 * BonsaiPoller::                
+* SvnSource::                   
 @end menu
 
 @node Choosing ChangeSources, CVSToys - PBService, Change Sources, Change Sources
@@ -2754,7 +2757,6 @@
 
 @end table
 
-
 @node P4Source, BonsaiPoller, PBChangeSource, Change Sources
 @subsection P4Source
 
@@ -2806,7 +2808,7 @@
 ))
 @end example
 
- at node BonsaiPoller,  , P4Source, Change Sources
+ at node BonsaiPoller, SvnSource, P4Source, Change Sources
 @subsection BonsaiPoller
 
 @csindex buildbot.changes.bonsaipoller.BonsaiPoller
@@ -2823,6 +2825,59 @@
 arguments it accepts.
 
 
+ at node SvnSource,  , BonsaiPoller, Change Sources
+ at subsection SvnSource
+
+ at csindex buildbot.changes.svnpoller.SvnSource
+
+The @code{SvnSource} periodically polls a
+ at uref{http://subversion.tigris.org/, Subversion} depot for changes, by
+running the @code{svn} command in a subshell. It accepts the following
+arguments:
+
+ at table @samp
+ at item @code{svnurl}
+The base URL path to watch. The repository this references could be
+local or remote, depending upon the URL's prefix.
+
+ at item @code{branch}
+The branch to watch. Defaults to '/trunk'.
+
+ at item @code{svnuser}
+The Subversion user.
+
+ at item @code{svnpasswd}
+The Subversion user's password.
+
+ at item @code{pollinterval}
+How often to poll, in seconds. Defaults to 600 (10 minutes).
+
+ at item @code{histmax}
+The maximum number of changes to inspect at a time. If more than this
+number occur since the last poll, older changes will be silently
+ignored. Larger values of histmax will cause more time and memory to
+be consumed on each poll attempt. @code{histmax} defaults to 100.
+ at end table
+
+ at heading Example
+
+This example watches the Subversion SVN tree itself for changes. It
+only pays attention to changes that appear on the trunk, and it labels
+all of its Change objects as being on branch=None (which is the
+convention within the buildbot for trunk changes).
+
+ at example
+from buildbot.changes.svnpoller import SvnSource
+ss = SvnSource('http://svn.collab.net/repos/svn/trunk')
+c['sources'].append(ss)
+ at end example
+
+This configuration uses the @code{SvnPORT}, @code{SvnUSER}, and
+ at code{SvnPASSWD} specified in the buildmaster's environment. It
+watches a project in which the branch name is simply the next path
+component, and the file is all path components after.
+
+
 @node Build Process, Status Delivery, Getting Source Code Changes, Top
 @chapter Build Process
 




From warner at users.sourceforge.net  Mon Oct  2 00:13:20 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:20 +0000
Subject: [Buildbot-commits] buildbot/buildbot/changes svnpoller.py,1.1,1.2
Message-ID: <E1GUBQu-0005CP-Og@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/changes
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17137/buildbot/changes

Modified Files:
	svnpoller.py 
Log Message:
[project @ svnpoller: lots of work, make branches behave properly, write lots of tests, docs]

Original author: warner at lothar.com
Date: 2006-10-01 23:23:25

Index: svnpoller.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/changes/svnpoller.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -d -r1.1 -r1.2
--- svnpoller.py	2 Oct 2006 00:13:11 -0000	1.1
+++ svnpoller.py	2 Oct 2006 00:13:18 -0000	1.2
@@ -2,12 +2,12 @@
 
 # Based on the work of Dave Peticolas for the P4poll
 # Changed to svn (using xml.dom.minidom) by Niklaus Giger
+# Hacked beyond recognition by Brian Warner
 
 import time
 
-from twisted.python import log, failure
-from twisted.internet import defer, reactor
-from twisted.internet.utils import getProcessOutput
+from twisted.python import log
+from twisted.internet import defer, reactor, utils
 from twisted.internet.task import LoopingCall
 
 from buildbot import util
@@ -31,21 +31,25 @@
     return (None, path)
 
 def split_file_branches(path):
+    # turn trunk/subdir/file.c into (None, "subdir/file.c")
+    # and branches/1.5.x/subdir/file.c into ("branches/1.5.x", "subdir/file.c")
     pieces = path.split('/')
     if pieces[0] == 'trunk':
         return (None, '/'.join(pieces[1:]))
     elif pieces[0] == 'branches':
-        return (pieces[1], '/'.join(pieces[2:]))
+        return ('/'.join(pieces[0:2]), '/'.join(pieces[2:]))
     else:
         return None
 
 
 class SvnSource(base.ChangeSource, util.ComparableMixin):
-    """This source will poll a perforce repository for changes and submit
+    """This source will poll a Subversion repository for changes and submit
     them to the change master."""
 
-    compare_attrs = ["svnuser", "svnpasswd", "svnurl",
-                     "pollinterval", "histmax"]
+    compare_attrs = ["svnurl", "split_file_function",
+                     "svnuser", "svnpasswd",
+                     "pollinterval", "histmax",
+                     "svnbin"]
 
     parent = None # filled in when we're added
     last_change = None
@@ -54,8 +58,8 @@
 
     def __init__(self, svnurl, split_file=None,
                  svnuser=None, svnpasswd=None,
-                 svnbin='svn',
-                 pollinterval=10*60, histmax=100):
+                 pollinterval=10*60, histmax=100,
+                 svnbin='svn'):
         """
         @type  svnurl: string
         @param svnurl: the SVN URL that describes the repository and
@@ -100,7 +104,8 @@
                             if pieces[0] == 'trunk':
                                 return (None, '/'.join(pieces[1:]))
                             elif pieces[0] == 'branches':
-                                return (pieces[1], '/'.join(pieces[2:]))
+                                return ('/'.join(pieces[0:2]),
+                                        '/'.join(pieces[2:]))
                             else:
                                 return None
 
@@ -115,7 +120,8 @@
                                 pieces.pop(0) # remove 'trunk'
                             elif pieces[0] == 'branches':
                                 pieces.pop(0) # remove 'branches'
-                                branch = pieces.pop(0) # grab branch name
+                                # grab branch name
+                                branch = 'branches/' + pieces.pop(0)
                             else:
                                 return None # something weird
                             projectname = pieces.pop(0)
@@ -141,11 +147,6 @@
         @type  svnpasswd:    string
         @param svnpasswd:    If set, the --password option will be added.
 
-        @type  svnbin:       string
-        @param svnbin:       path to svn binary, defaults to just 'svn'. Use
-                             this if your subversion command lives in an
-                             unusual location.
-
         @type  pollinterval: int
         @param pollinterval: interval in seconds between polls. The default
                              is 600 seconds (10 minutes). Smaller values
@@ -159,6 +160,11 @@
                              system load, but if more than histmax changes
                              are recorded between polls, the extra ones will
                              be silently lost.
+
+        @type  svnbin:       string
+        @param svnbin:       path to svn binary, defaults to just 'svn'. Use
+                             this if your subversion command lives in an
+                             unusual location.
         """
 
         if svnurl.endswith("/"):
@@ -171,7 +177,8 @@
         self.svnbin = svnbin
         self.pollinterval = pollinterval
         self.histmax = histmax
-        self._root = None
+        self._prefix = None
+        self.overrun_counter = 0
         self.loop = LoopingCall(self.checksvn)
 
     def split_file(self, path):
@@ -181,6 +188,7 @@
         return f(path)
 
     def startService(self):
+        log.msg("SvnSource(%s) starting" % self.svnurl)
         base.ChangeSource.startService(self)
         # Don't start the loop just yet because the reactor isn't running.
         # Give it a chance to go and install our SIGCHLD handler before
@@ -188,11 +196,12 @@
         reactor.callLater(0, self.loop.start, self.pollinterval)
 
     def stopService(self):
+        log.msg("SvnSource(%s) shutting down" % self.svnurl)
         self.loop.stop()
         return base.ChangeSource.stopService(self)
 
     def describe(self):
-        return "svnsource %s branch %s" % ( self.svnurl, self.branch)
+        return "SvnSource watching %s" % self.svnurl
 
     def checksvn(self):
         # Our return value is only used for unit testing.
@@ -226,43 +235,53 @@
 
         # whew.
 
+        if self.working:
+            log.msg("SvnSource(%s) overrun: timer fired but the previous "
+                    "poll had not yet finished.")
+            self.overrun_counter += 1
+            return defer.succeed(None)
+        self.working = True
+
+        log.msg("SvnSource polling")
         if not self._prefix:
-            # this sets self._prefix when it finishes
-            d = self._determine_prefix()
+            # this sets self._prefix when it finishes. It fires with
+            # self._prefix as well, because that makes the unit tests easier
+            # to write.
+            d = self.get_root()
+            d.addCallback(self.determine_prefix)
         else:
-            d = defer.succeed(None)
+            d = defer.succeed(self._prefix)
 
-        if self.working:
-            dbgMsg("Skipping checksvn because last one has not finished")
-            return d
+        d.addCallback(self.get_logs)
+        d.addCallback(self.parse_logs)
+        d.addCallback(self.get_new_logentries)
+        d.addCallback(self.create_changes)
+        d.addCallback(self.submit_changes)
+        d.addBoth(self.finished)
+        return d
 
-        self.working = True
-        d.addCallback(lambda res: self._get_logs())
-        d.addCallback(self._parse_logs)
-        d.addCallback(self._get_new_logentries)
-        d.addCallback(self._create_changes)
-        d.addCallback(self._submit_changes)
-        d.addBoth(self._finished)
+    def getProcessOutput(self, args):
+        # this exists so we can override it during the unit tests
+        d = utils.getProcessOutput(self.svnbin, args, {})
         return d
 
-    def _determine_prefix(self):
+    def get_root(self):
         args = ["info", "--xml", "--non-interactive", self.svnurl]
         if self.svnuser:
             args.extend(["--username=%s" % self.svnuser])
         if self.svnpasswd:
             args.extend(["--password=%s" % self.svnpasswd])
-        d = getProcessOutput(self.svnbin, args, {})
-        d.addCallback(self._determine_prefix_2)
+        d = self.getProcessOutput(args)
         return d
 
-    def _determine_prefix_2(self, output):
-	try:
-	    doc = xml.dom.minidom.parseString(output)
-	except xml.parsers.expat.ExpatError:
-	    dbgMsg("_process_changes: ExpatError in %s" % output)
+    def determine_prefix(self, output):
+        try:
+            doc = xml.dom.minidom.parseString(output)
+        except xml.parsers.expat.ExpatError:
+            dbgMsg("_process_changes: ExpatError in %s" % output)
             log.msg("SvnSource._determine_prefix_2: ExpatError in '%s'"
                     % output)
-	    raise
+            raise
         rootnodes = doc.getElementsByTagName("root")
         if not rootnodes:
             # this happens if the URL we gave was already the root. In this
@@ -282,7 +301,7 @@
                 (self.svnurl, root, self._prefix))
         return self._prefix
 
-    def _get_logs(self):
+    def get_logs(self, ignored_prefix=None):
         args = []
         args.extend(["log", "--xml", "--verbose", "--non-interactive"])
         if self.svnuser:
@@ -290,22 +309,21 @@
         if self.svnpasswd:
             args.extend(["--password=%s" % self.svnpasswd])
         args.extend(["--limit=%d" % (self.histmax), self.svnurl])
-        env = {}
-        res = getProcessOutput(self.svnbin, args, env)
-	log.msg("_get_changes args %s end %s returns %s" % (args, env, res))
-	return res
+        d = self.getProcessOutput(args)
+        return d
 
-    def _parse_logs(self, output):
+    def parse_logs(self, output):
         # parse the XML output, return a list of <logentry> nodes
-	try:
-	    doc = xml.dom.minidom.parseString(output)
-	except xml.parsers.expat.ExpatError:
-	    dbgMsg("_process_changes: ExpatError in %s" % output)
+        try:
+            doc = xml.dom.minidom.parseString(output)
+        except xml.parsers.expat.ExpatError:
+            dbgMsg("_process_changes: ExpatError in %s" % output)
             log.msg("SvnSource._parse_changes: ExpatError in '%s'" % output)
             raise
         logentries = doc.getElementsByTagName("logentry")
         return logentries
 
+
     def _filter_new_logentries(self, logentries, last_change):
         # given a list of logentries, return a tuple of (new_last_change,
         # new_logentries), where new_logentries contains only the ones after
@@ -314,30 +332,40 @@
             # no entries, so last_change must stay at None
             return (None, [])
 
-	mostRecent = int(logentries[0].getAttribute("revision"))
+        mostRecent = int(logentries[0].getAttribute("revision"))
 
         if last_change is None:
             # if this is the first time we've been run, ignore any changes
             # that occurred before now. This prevents a build at every
             # startup.
-	    log.msg('svnPoller: starting at change %s' % mostRecent)
-	    self.last_change = mostRecent
-	    return (mostRecent, [])
+            log.msg('svnPoller: starting at change %s' % mostRecent)
+            return (mostRecent, [])
 
-	if last_change == mostRecent:
+        if last_change == mostRecent:
             # an unmodified repository will hit this case
-	    log.msg('svnPoller: _process_changes last %s mostRecent %s' % (
-		      last_change, mostRecent))
+            log.msg('svnPoller: _process_changes last %s mostRecent %s' % (
+                      last_change, mostRecent))
             return (mostRecent, [])
 
         new_logentries = []
-	for el in logentries:
+        for el in logentries:
             if last_change == int(el.getAttribute("revision")):
-		break
+                break
             new_logentries.append(el)
         new_logentries.reverse() # return oldest first
         return (mostRecent, new_logentries)
 
+    def get_new_logentries(self, logentries):
+        last_change = self.last_change
+        (new_last_change,
+         new_logentries) = self._filter_new_logentries(logentries,
+                                                       self.last_change)
+        self.last_change = new_last_change
+        log.msg('svnPoller: _process_changes %s .. %s' %
+                (last_change, new_last_change))
+        return new_logentries
+
+
     def _get_text(self, element, tag_name):
         child_nodes = element.getElementsByTagName(tag_name)[0].childNodes
         text = "".join([t.data for t in child_nodes])
@@ -350,41 +378,39 @@
         relative_path = path[len(self._prefix):]
         if relative_path.startswith("/"):
             relative_path = relative_path[1:]
-        branch, final_path = self.split_file(relative_path)
-        return branch, final_path
-
-    def _get_new_logentries(self, logentries):
-        last_change = self.last_change
-        (new_last_change,
-         new_logentries) = self._filter_new_logentries(logentries,
-                                                       self.last_change)
-        self.last_change = new_last_change
-	log.msg('svnPoller: _process_changes %s .. %s' %
-                (last_change, new_last_change))
-        return new_logentries
+        where = self.split_file(relative_path)
+        # 'where' is either None or (branch, final_path)
+        return where
 
-    def _create_changes(self, new_logentries):
+    def create_changes(self, new_logentries):
         changes = []
 
         for el in new_logentries:
-	    branch_files = [] # get oldest change first
-	    revision = int(el.getAttribute("revision"))
-	    dbgMsg("Adding change revision %s" % (revision,))
-	    author   = self._get_text(el, "author")
-	    comments = self._get_text(el, "msg")
-	    when     = self._get_text(el, "date")
-	    when     = time.mktime(time.strptime("%.19s" % when,
+            branch_files = [] # get oldest change first
+            # TODO: revisit this, I think I've settled on Change.revision
+            # being a string everywhere, and leaving the interpretation
+            # of that string up to b.s.source.SVN methods
+            revision = int(el.getAttribute("revision"))
+            dbgMsg("Adding change revision %s" % (revision,))
+            author   = self._get_text(el, "author")
+            comments = self._get_text(el, "msg")
+            when     = self._get_text(el, "date")
+            when     = time.mktime(time.strptime("%.19s" % when,
                                                  "%Y-%m-%dT%H:%M:%S"))
             branches = {}
-	    pathlist = el.getElementsByTagName("paths")[0]
-	    for p in pathlist.getElementsByTagName("path"):
-		path = "".join([t.data for t in p.childNodes])
+            pathlist = el.getElementsByTagName("paths")[0]
+            for p in pathlist.getElementsByTagName("path"):
+                path = "".join([t.data for t in p.childNodes])
                 if path.startswith("/"):
                     path = path[1:]
-                branch, filename = self._transform_path(path)
-                if not branch in branches:
-                    branches[branch] = []
-                branches[branch].append(filename)
+                where = self._transform_path(path)
+                # if 'where' is None, the file was outside any project that
+                # we care about and we should ignore it
+                if where:
+                    branch, filename = where
+                    if not branch in branches:
+                        branches[branch] = []
+                    branches[branch].append(filename)
 
             for branch in branches:
                 c = Change(who=author,
@@ -397,17 +423,13 @@
 
         return changes
 
-    def _submit_changes(self, changes):
+    def submit_changes(self, changes):
         for c in changes:
-	    self.parent.addChange(c)
+            self.parent.addChange(c)
 
-    def _finished(self, res):
+    def finished(self, res):
+        log.msg("SvnSource finished polling")
         dbgMsg('_finished : %s' % res)
         assert self.working
         self.working = False
-
-        # Again, the return value is only for unit testing.
-        # If there's a failure, log it so it isn't lost.
-        if isinstance(res, failure.Failure):
-            dbgMsg('checksvn failed: %s' % res)
         return res




From warner at users.sourceforge.net  Mon Oct  2 00:13:20 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:20 +0000
Subject: [Buildbot-commits] buildbot/docs buildbot.texinfo,1.82,1.83
Message-ID: <E1GUBQu-0005CX-Qt@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/docs
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17137/docs

Modified Files:
	buildbot.texinfo 
Log Message:
[project @ svnpoller: lots of work, make branches behave properly, write lots of tests, docs]

Original author: warner at lothar.com
Date: 2006-10-01 23:23:25

Index: buildbot.texinfo
===================================================================
RCS file: /cvsroot/buildbot/buildbot/docs/buildbot.texinfo,v
retrieving revision 1.82
retrieving revision 1.83
diff -u -d -r1.82 -r1.83
--- buildbot.texinfo	2 Oct 2006 00:13:12 -0000	1.82
+++ buildbot.texinfo	2 Oct 2006 00:13:18 -0000	1.83
@@ -2830,52 +2830,274 @@
 
 @csindex buildbot.changes.svnpoller.SvnSource
 
-The @code{SvnSource} periodically polls a
- at uref{http://subversion.tigris.org/, Subversion} depot for changes, by
-running the @code{svn} command in a subshell. It accepts the following
-arguments:
+The @code{buildbot.changes.svnpoller.SvnSource} is a ChangeSource
+which periodically polls a @uref{http://subversion.tigris.org/,
+Subversion} repository for new revisions, by running the @code{svn
+log} command in a subshell. It can watch a single branch or multiple
+branches.
 
- at table @samp
- at item @code{svnurl}
-The base URL path to watch. The repository this references could be
-local or remote, depending upon the URL's prefix.
+ at code{SvnSource} accepts the following arguments:
 
- at item @code{branch}
-The branch to watch. Defaults to '/trunk'.
+ at table @code
+ at item svnurl
+The base URL path to watch, like
+ at code{svn://svn.twistedmatrix.com/svn/Twisted/trunk}, or
+ at code{http://divmod.org/svn/Divmod/}, or even
+ at code{file:///home/svn/Repository/ProjectA/branches/1.5/}. This must
+include the access scheme, the location of the repository (both the
+hostname for remote ones, and any additional directory names necessary
+to get to the repository), and the sub-path within the repository's
+virtual filesystem for the project and branch of interest.
 
- at item @code{svnuser}
-The Subversion user.
+The @code{SvnSource} will only pay attention to files inside the
+subdirectory specified by the complete svnurl.
 
- at item @code{svnpasswd}
-The Subversion user's password.
+ at item split_file
+A function to convert pathnames into (branch, relative_pathname)
+tuples. Use this to explain your repository's branch-naming policy to
+ at code{SvnSource}. This function must accept a single string and return
+a two-entry tuple. There are a few utility functions in
+ at code{buildbot.changes.svnpoller} that can be used as a
+ at code{split_file} function, see below for details.
 
- at item @code{pollinterval}
-How often to poll, in seconds. Defaults to 600 (10 minutes).
+The default value always returns (None, path), which indicates that
+all files are on the trunk.
+
+Subclasses of @code{SvnSource} can override the @code{split_file}
+method instead of using the @code{split_file=} argument.
+
+ at item svnuser
+An optional string parameter. If set, the @code{--user} argument will
+be added to all @code{svn} commands. Use this if you have to
+authenticate to the svn server before you can do @code{svn info} or
+ at code{svn log} commands.
+
+ at item svnpasswd
+Like @code{svnuser}, this will cause a @code{--password} argument to
+be passed to all svn commands.
+
+ at item pollinterval
+How often to poll, in seconds. Defaults to 600 (checking once every 10
+minutes). Lower this if you want the buildbot to notice changes
+faster, raise it if you want to reduce the network and CPU load on
+your svn server. Please be considerate of public SVN repositories by
+using a large interval when polling them.
+
+ at item histmax
+The maximum number of changes to inspect at a time. Every POLLINTERVAL
+seconds, the @code{SvnSource} asks for the last HISTMAX changes and
+looks through them for any ones it does not already know about. If
+more than HISTMAX revisions have been committed since the last poll,
+older changes will be silently ignored. Larger values of histmax will
+cause more time and memory to be consumed on each poll attempt.
+ at code{histmax} defaults to 100.
+
+ at item svnbin
+This controls the @code{svn} executable to use. If subversion is
+installed in a weird place on your system (outside of the
+buildmaster's @code{$PATH}), use this to tell @code{SvnSource} where
+to find it. The default value of ``svn'' will almost always be
+sufficient.
 
- at item @code{histmax}
-The maximum number of changes to inspect at a time. If more than this
-number occur since the last poll, older changes will be silently
-ignored. Larger values of histmax will cause more time and memory to
-be consumed on each poll attempt. @code{histmax} defaults to 100.
 @end table
 
- at heading Example
+ at heading Branches
 
-This example watches the Subversion SVN tree itself for changes. It
-only pays attention to changes that appear on the trunk, and it labels
-all of its Change objects as being on branch=None (which is the
-convention within the buildbot for trunk changes).
+Each source file that is tracked by a Subversion repository has a
+fully-qualified SVN URL in the following form:
+(REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the
+ at code{SvnSource}, you give it a @code{svnurl} value that includes all
+of the REPOURL and possibly some portion of the PROJECT-plus-BRANCH
+string. The @code{SvnSource} is responsible for producing Changes that
+contain a branch name and a FILEPATH (which is relative to the top of
+a checked-out tree). The details of how these strings are split up
+depend upon how your repository names its branches.
+
+ at subheading PROJECT/BRANCHNAME/FILEPATH repositories
+
+One common layout is to have all the various projects that share a
+repository get a single top-level directory each. Then under a given
+project's directory, you get two subdirectories, one named ``trunk''
+and another named ``branches''. Under ``branches'' you have a bunch of
+other directories, one per branch, with names like ``1.5.x'' and
+``testing''. It is also common to see directories like ``tags'' and
+``releases'' next to ``branches'' and ``trunk''.
+
+For example, the Twisted project has a subversion server on
+``svn.twistedmatrix.com'' that hosts several sub-projects. The
+repository is available through a SCHEME of ``svn:''. The primary
+sub-project is Twisted, of course, with a repository root of
+``svn://svn.twistedmatrix.com/svn/Twisted''. Another sub-project is
+Informant, with a root of
+``svn://svn.twistedmatrix.com/svn/Informant'', etc. Inside any
+checked-out Twisted tree, there is a file named bin/trial (which is
+used to run unit test suites).
+
+The trunk for Twisted is in
+``svn://svn.twistedmatrix.com/svn/Twisted/trunk'', and the
+fully-qualified SVN URL for the trunk version of @code{trial} would be
+``svn://svn.twistedmatrix.com/svn/Twisted/trunk/bin/trial''. The same
+SVNURL for that file on a branch named ``1.5.x'' would be
+``svn://svn.twistedmatrix.com/svn/Twisted/branches/1.5.x/bin/trial''.
+
+To set up a @code{SvnSource} that watches the Twisted trunk (and
+nothing else), we would use the following:
 
 @example
 from buildbot.changes.svnpoller import SvnSource
-ss = SvnSource('http://svn.collab.net/repos/svn/trunk')
+s = SvnSource("svn://svn.twistedmatrix.com/svn/Twisted/trunk")
 c['sources'].append(ss)
 @end example
 
-This configuration uses the @code{SvnPORT}, @code{SvnUSER}, and
- at code{SvnPASSWD} specified in the buildmaster's environment. It
-watches a project in which the branch name is simply the next path
-component, and the file is all path components after.
+In this case, every Change that our @code{SvnSource} produces will
+have @code{.branch=None}, to indicate that the Change is on the trunk.
+No other sub-projects or branches will be tracked.
+
+If we want our ChangeSource to follow multiple branches, we have to do
+two things. First we have to change our @code{svnurl=} argument to
+watch more than just ``.../Twisted/trunk''. We will set it to
+``.../Twisted'' so that we'll see both the trunk and all the branches.
+Second, we have to tell @code{SvnSource} how to split the
+(PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository
+out into (BRANCH) and (FILEPATH) pairs.
+
+We do the latter by providing a ``split_file'' function. This function
+is responsible for splitting something like
+``branches/1.5.x/bin/trial'' into @code{branch}=''branches/1.5.x'' and
+ at code{filepath}=''bin/trial''. This function is always given a string
+that names a file relative to the subdirectory pointed to by the
+ at code{SvnSource}'s @code{svnurl=} argument. It is expected to return a
+(BRANCHNAME, FILEPATH) tuple (in which FILEPATH is relative to the
+branch indicated), or None to indicate that the file is outside any
+project of interest.
+
+(note that we want to see ``branches/1.5.x'' rather than just
+``1.5.x'' because when we perform the SVN checkout, we will probably
+append the branch name to the baseURL, which requires that we keep the
+``branches'' component in there. Other VC schemes use a different
+approach towards branches and may not require this artifact.)
+
+If your repository uses this same PROJECT/BRANCH/FILEPATH naming
+scheme, the following function will work:
+
+ at example
+def split_file_branches(path):
+    pieces = path.split('/')
+    if pieces[0] == 'trunk':
+        return (None, '/'.join(pieces[1:]))
+    elif pieces[0] == 'branches':
+        return ('/'.join(pieces[0:2]),
+                '/'.join(pieces[2:]))
+    else:
+        return None
+ at end example
+
+This function is provided as
+ at code{buildbot.changes.svnpoller.split_file_branches} for your
+convenience. So to have our Twisted-watching @code{SvnSource} follow
+multiple branches, we would use this:
+
+ at example
+from buildbot.changes.svnpoller import SvnSource, split_file_branches
+s = SvnSource("svn://svn.twistedmatrix.com/svn/Twisted",
+              split_file=split_file_branches)
+c['sources'].append(ss)
+ at end example
+
+Changes for all sorts of branches (with names like ``branches/1.5.x'',
+and None to indicate the trunk) will be delivered to the Schedulers.
+Each Scheduler is then free to use or ignore each branch as it sees
+fit.
+
+ at subheading BRANCHNAME/PROJECT/FILEPATH repositories
+
+Another common way to organize a Subversion repository is to put the
+branch name at the top, and the projects underneath. This is
+especially frequent when there are a number of related sub-projects
+that all get released in a group.
+
+For example, Divmod.org hosts a project named ``Nevow'' as well as one
+named ``Quotient''. In a checked-out Nevow tree there is a directory
+named ``formless'' that contains a python source file named
+``webform.py''. This repository is accessible via webdav (and thus
+uses an ``http:'' scheme) through the divmod.org hostname. There are
+many branches in this repository, and they use a
+(BRANCHNAME)/(PROJECT) naming policy.
+
+The fully-qualified SVN URL for the trunk version of webform.py is
+ at code{http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py}.
+You can do an @code{svn co} with that URL and get a copy of the latest
+version. The 1.5.x branch version of this file would have a URL of
+ at code{http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py}.
+The whole Nevow trunk would be checked out with
+ at code{http://divmod.org/svn/Divmod/trunk/Nevow}, while the Quotient
+trunk would be checked out using
+ at code{http://divmod.org/svn/Divmod/trunk/Quotient}.
+
+Now suppose we want to have an @code{SvnSource} that only cares about
+the Nevow trunk. This case looks just like the PROJECT/BRANCH layout
+described earlier:
+
+ at example
+from buildbot.changes.svnpoller import SvnSource
+s = SvnSource("http://divmod.org/svn/Divmod/trunk/Nevow")
+c['sources'].append(ss)
+ at end example
+
+But what happens when we want to track multiple Nevow branches? We
+have to point our @code{svnurl=} high enough to see all those
+branches, but we also don't want to include Quotient changes (since
+we're only building Nevow). To accomplish this, we must rely upon the
+ at code{split_file} function to help us tell the difference between
+files that belong to Nevow and those that belong to Quotient, as well
+as figuring out which branch each one is on.
+
+ at example
+from buildbot.changes.svnpoller import SvnSource
+s = SvnSource("http://divmod.org/svn/Divmod",
+              split_file=my_file_splitter)
+c['sources'].append(ss)
+ at end example
+
+The @code{my_file_splitter} function will be called with
+repository-relative pathnames like:
+
+ at table @code
+ at item trunk/Nevow/formless/webform.py
+This is a Nevow file, on the trunk. We want the Change that includes this
+to see a filename of @code{formless/webform.py"}, and a branch of None
+
+ at item branches/1.5.x/Nevow/formless/webform.py
+This is a Nevow file, on a branch. We want to get
+branch=''branches/1.5.x'' and filename=''formless/webform.py''.
+
+ at item trunk/Quotient/setup.py
+This is a Quotient file, so we want to ignore it by having
+ at code{my_file_splitter} return None.
+
+ at item branches/1.5.x/Quotient/setup.py
+This is also a Quotient file, which should be ignored.
+ at end table
+
+The following definition for @code{my_file_splitter} will do the job:
+
+ at example
+def my_file_splitter(path):
+    pieces = path.split('/')
+    if pieces[0] == 'trunk':
+        branch = None
+        pieces.pop(0) # remove 'trunk'
+    elif pieces[0] == 'branches':
+        pieces.pop(0) # remove 'branches'
+        # grab branch name
+        branch = 'branches/' + pieces.pop(0)
+    else:
+        return None # something weird
+    projectname = pieces.pop(0)
+    if projectname != 'Nevow':
+        return None # wrong project
+    return (branch, '/'.join(pieces))
+ at end example
 
 
 @node Build Process, Status Delivery, Getting Source Code Changes, Top




From warner at users.sourceforge.net  Mon Oct  2 00:13:20 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:20 +0000
Subject: [Buildbot-commits] buildbot/buildbot/test test_svnpoller.py, 1.1,
	1.2
Message-ID: <E1GUBQu-0005CM-OI@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/test
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17137/buildbot/test

Modified Files:
	test_svnpoller.py 
Log Message:
[project @ svnpoller: lots of work, make branches behave properly, write lots of tests, docs]

Original author: warner at lothar.com
Date: 2006-10-01 23:23:25

Index: test_svnpoller.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_svnpoller.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -d -r1.1 -r1.2
--- test_svnpoller.py	2 Oct 2006 00:13:12 -0000	1.1
+++ test_svnpoller.py	2 Oct 2006 00:13:18 -0000	1.2
@@ -1,227 +1,9 @@
-# Here are the tests for the SvnSource
+# -*- test-case-name: buildbot.test.test_svnpoller -*-
 
-import sys
 import time
-
-from twisted.python import log, failure
 from twisted.internet import defer
 from twisted.trial import unittest
-
-from buildbot.twcompat import maybeWait
-from buildbot.changes.changes import Change
-from buildbot.changes.svnpoller import SvnSource, split_file_branches
-
-# was changes 1012 in xenomai.org
-svn_change_1 = """<?xml version="1.0" encoding="utf-8"?>
-<log>
-<logentry
-   revision="101">
-<author>rpm</author>
-<date>2006-05-17T14:58:28.494960Z</date>
-<paths>
-<path
-   action="M">/branch/ksrc/arch/i386/hal.c</path>
-</paths>
-<msg>Remove unused variable</msg>
-</logentry>
-</log>
-"""
-
-svn_change_2 = """<?xml version="1.0" encoding="utf-8"?>
-<log>
-<logentry
-   revision="102">
-<author>rpm</author>
-<date>2006-05-15T12:54:08.891420Z</date>
-<paths>
-<path
-   action="M">/trunk/ChangeLog</path>
-<path
-   action="D">/trunk/ksrc/first_file</path>
-<path
-   action="D">/trunk/ksrc/second_file</path>
-</paths>
-<msg>Initial Adeos support</msg>
-</logentry>
-</log>
-"""
-
-svn_change_3 = """<?xml version="1.0" encoding="utf-8"?>
-<log>
-<logentry
-   revision="102">
-<author>rpm</author>
-<date>2006-05-15T12:54:08.891420Z</date>
-<paths>
-<path
-   action="M">/trunk/ChangeLog</path>
-<path
-   action="D">/trunk/ksrc/first_file</path>
-<path
-   action="D">/trunk/ksrc/second_file</path>
-</paths>
-<msg>Upgrade Adeos support</msg>
-</logentry>
-</log>
-"""
-
-def dbgMsg(myString):
-    log.msg(myString)
-    return 1
-
-class MockSvnSource(SvnSource):
-    """Test SvnSource which doesn't actually invoke svn."""
-    invocation = 0
-
-    def __init__(self, svnchanges, *args, **kwargs):
-        SvnSource.__init__(self, None, *args, **kwargs)
-        self.svnchanges = svnchanges
-
-    def _get_changes(self):
-        assert self.working
-        result = self.svnchanges[self.invocation]
-        self.invocation += 1
-        #log.msg("MockSvnSource._get_changes %s result %s " % (self.invocation-1, result))
-        dbgMsg("MockSvnSource._get_changes %s " % (self.invocation-1))
-        return defer.succeed(result)
-
-    def _get_describe(self, dummy, num):
-        assert self.working
-        dbgMsg("MockSvnSource._get_describe %s " % num)
-        return defer.succeed(self.svnchanges[num])
-
-class TestSvnPoller(unittest.TestCase):
-    def setUp(self):
-        self.changes = []
-        self.addChange = self.changes.append
-
-    def failUnlessIn(self, substr, string):
-        # this is for compatibility with python2.2
-        if isinstance(string, str):
-            self.failUnless(string.find(substr) != -1)
-        else:
-            self.assertIn(substr, string)
-
-    def testCheck(self):
-        """successful checks"""
-        self.t = MockSvnSource(svnchanges=[ svn_change_1, svn_change_2, svn_change_3],
-                              svnuser=None,
-                              svnurl='/trunk/',)
-        self.t.parent = self
-
-        # The first time, it just learns the change to start at
-        self.assert_(self.t.last_change is None)
-        self.assert_(not self.t.working)
-	dbgMsg("zzz")
-        return maybeWait(self.t.checksvn().addCallback(self._testCheck2))
-
-    def _testCheck2(self, res):
-	dbgMsg("zzz1")
-        self.assertEquals(self.changes, [])
-	dbgMsg("zzz2 %s %s" % (self.t.last_change, self.changes))
-        self.assertEquals(self.t.last_change, '101')
-	dbgMsg("zzz3")
-
-        # Subsequent times, it returns Change objects for new changes.
-        return self.t.checksvn().addCallback(self._testCheck3)
-
-    def _testCheck3(self, res):
-        # They're supposed to go oldest to newest, so this one must be first.
-	tstChange1 = Change(who='rpm',
-                   files=['/trunk/ChangeLog',
-		          '/trunk/ksrc/first_file',
-			  '/trunk/ksrc/second_file'],
-                   comments="Initial Adeos support",
-                   revision='2',
-                   when=self.makeTime("2006/05/15 12:54:08"),
-                   branch='trunk').asText()
-	dbgMsg("tstChange" + tstChange1)
-	dbgMsg("changes[0]" + self.changes[0].asText())
-
-	self.assertEquals(self.changes[0].asText(), tstChange1)
-        # Subsequent times, it returns Change objects for new changes.
-        return self.t.checksvn().addCallback(self._testCheck4)
-
-    def _testCheck4(self, res):
-	dbgMsg("zzz5 %s " % len(self.changes))
-        self.assertEquals(len(self.changes), 1)
-	dbgMsg("zzz6 %s %s" % (self.t.last_change, self.changes))
-        self.assertEquals(self.t.last_change, '102')
-	dbgMsg("zzz7")
-        self.assert_(not self.t.working)
-	tstChange2 = Change(who='rpm',
-                   files=['/trunk/ChangeLog',
-		          '/trunk/ksrc/first_file',
-			  '/trunk/ksrc/second_file'],
-                   comments="Initial Adeos support",
-                   revision='2',
-                   when=self.makeTime("2006/05/15 12:54:08"),
-                   branch='trunk').asText()
-	dbgMsg("changes[0]" + self.changes[0].asText())
-	dbgMsg("tstChange2" + tstChange2)
-        self.assertEquals(self.changes[0].asText(), tstChange2)
-	dbgMsg(7777)
-
-    def makeTime(self, timestring):
-        datefmt = '%Y/%m/%d %H:%M:%S'
-        when = time.mktime(time.strptime(timestring, datefmt))
-        return when
-
-    def testFailedChanges(self):
-        """'svn changes' failure is properly reported"""
-        self.t = MockSvnSource(svnchanges=['Subversion client error:\n...'],
-                               svnuser=None,
-                               svnurl="/trunk")
-        self.t.parent = self
-        d = self.t.checksvn()
-        d.addBoth(self._testFailedChanges2)
-        return maybeWait(d)
-
-    def _testFailedChanges2(self, f):
-        self.assert_(isinstance(f, failure.Failure))
-        self.failUnlessIn('Subversion client error', str(f))
-        self.assert_(not self.t.working)
-
-    def testFailedDescribe(self):
-        """'svn describe' failure is properly reported"""
-        self.t = MockSvnSource(svnchanges=[
-	                       svn_change_1,
-			       'Subversion client error:\n...',
-			       svn_change_2,],
-                               svnuser=None)
-        self.t.parent = self
-        d = self.t.checksvn()
-	dbgMsg("xxx")
-        d.addCallback(self._testFailedDescribe2)
-        return maybeWait(d)
-
-    def _testFailedDescribe2(self, res):
-        # first time finds nothing; check again.
-	dbgMsg("yy")
-        res = self.t.checksvn().addBoth(self._testFailedDescribe3)
-        return res
-
-    def _testFailedDescribe3(self, f):
-	dbgMsg("yy1 %s" % f)
-        self.assert_(isinstance(f, failure.Failure))
-	dbgMsg("yy2")
-        self.failUnlessIn('Subversion client error', str(f))
-	dbgMsg("yy3")
-        self.assert_(not self.t.working)
-	dbgMsg("yy4")
-        self.assertEquals(self.t.last_change, '101')
-	dbgMsg("yy5")
-
-    def testAlreadyWorking(self):
-        """don't launch a new poll while old is still going"""
-        self.t = SvnSource()
-        self.t.working = True
-        self.assert_(self.t.last_change is None)
-        d = self.t.checksvn()
-        d.addCallback(self._testAlreadyWorking2)
-
-    def _testAlreadyWorking2(self, res):
-        self.assert_(self.t.last_change is None)
+from buildbot.changes.svnpoller import SvnSource
 
 # this is the output of "svn info --xml
 # svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk"
@@ -307,7 +89,7 @@
         base = "svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk"
         s = SvnSource(base + "/")
         self.failUnlessEqual(s.svnurl, base) # certify slash-stripping
-        prefix = s._determine_prefix_2(prefix_output)
+        prefix = s.determine_prefix(prefix_output)
         self.failUnlessEqual(prefix, "trunk")
         self.failUnlessEqual(s._prefix, prefix)
 
@@ -315,42 +97,59 @@
         base = "svn+ssh://svn.twistedmatrix.com/svn/Twisted"
         s = SvnSource(base)
         self.failUnlessEqual(s.svnurl, base)
-        prefix = s._determine_prefix_2(prefix_output_2)
+        prefix = s.determine_prefix(prefix_output_2)
         self.failUnlessEqual(prefix, "")
 
     def test3(self):
         base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository"
         s = SvnSource(base)
         self.failUnlessEqual(s.svnurl, base)
-        prefix = s._determine_prefix_2(prefix_output_3)
+        prefix = s.determine_prefix(prefix_output_3)
         self.failUnlessEqual(prefix, "")
 
     def test4(self):
         base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample/trunk"
         s = SvnSource(base)
         self.failUnlessEqual(s.svnurl, base)
-        prefix = s._determine_prefix_2(prefix_output_4)
+        prefix = s.determine_prefix(prefix_output_4)
         self.failUnlessEqual(prefix, "sample/trunk")
 
 # output from svn log on .../SVN-Repository/sample
 # (so it includes trunk and branches)
-changes_output_1 = """\
-<?xml version="1.0"?>
-<log>
+sample_base = "file:///usr/home/warner/stuff/Projects/BuildBot/trees/misc/_trial_temp/test_vc/repositories/SVN-Repository/sample"
+sample_logentries = [None] * 4
+
+sample_logentries[3] = """\
+<logentry
+   revision="4">
+<author>warner</author>
+<date>2006-10-01T19:35:16.165664Z</date>
+<paths>
+<path
+   action="M">/sample/trunk/version.c</path>
+</paths>
+<msg>revised_to_2</msg>
+</logentry>
+"""
+
+sample_logentries[2] = """\
 <logentry
    revision="3">
 <author>warner</author>
-<date>2006-10-01T07:37:04.182499Z</date>
+<date>2006-10-01T19:35:10.215692Z</date>
 <paths>
 <path
    action="M">/sample/branch/main.c</path>
 </paths>
 <msg>commit_on_branch</msg>
 </logentry>
+"""
+
+sample_logentries[1] = """\
 <logentry
    revision="2">
 <author>warner</author>
-<date>2006-10-01T07:37:03.175326Z</date>
+<date>2006-10-01T19:35:09.154973Z</date>
 <paths>
 <path
    copyfrom-path="/sample/trunk"
@@ -359,44 +158,272 @@
 </paths>
 <msg>make_branch</msg>
 </logentry>
-</log>
 """
 
+sample_logentries[0] = """\
+<logentry
+   revision="1">
+<author>warner</author>
+<date>2006-10-01T19:35:08.642045Z</date>
+<paths>
+<path
+   action="A">/sample</path>
+<path
+   action="A">/sample/trunk</path>
+<path
+   action="A">/sample/trunk/subdir/subdir.c</path>
+<path
+   action="A">/sample/trunk/main.c</path>
+<path
+   action="A">/sample/trunk/version.c</path>
+<path
+   action="A">/sample/trunk/subdir</path>
+</paths>
+<msg>sample_project_files</msg>
+</logentry>
+"""
+
+sample_info_output = """\
+<?xml version="1.0"?>
+<info>
+<entry
+   kind="dir"
+   path="sample"
+   revision="4">
+<url>file:///usr/home/warner/stuff/Projects/BuildBot/trees/misc/_trial_temp/test_vc/repositories/SVN-Repository/sample</url>
+<repository>
+<root>file:///usr/home/warner/stuff/Projects/BuildBot/trees/misc/_trial_temp/test_vc/repositories/SVN-Repository</root>
+<uuid>4f94adfc-c41e-0410-92d5-fbf86b7c7689</uuid>
+</repository>
+<commit
+   revision="4">
+<author>warner</author>
+<date>2006-10-01T19:35:16.165664Z</date>
+</commit>
+</entry>
+</info>
+"""
+
+
+changes_output_template = """\
+<?xml version="1.0"?>
+<log>
+%s</log>
+"""
+
+def make_changes_output(maxrevision):
+    # return what 'svn log' would have just after the given revision was
+    # committed
+    logs = sample_logentries[0:maxrevision]
+    assert len(logs) == maxrevision
+    logs.reverse()
+    output = changes_output_template % ("".join(logs))
+    return output
+
+def split_file(path):
+    pieces = path.split("/")
+    if pieces[0] == "branch":
+        return "branch", "/".join(pieces[1:])
+    if pieces[0] == "trunk":
+        return None, "/".join(pieces[1:])
+    raise RuntimeError("there shouldn't be any files like %s" % path)
+
+class MySvnSource(SvnSource):
+    def __init__(self, *args, **kwargs):
+        SvnSource.__init__(self, *args, **kwargs)
+        self.pending_commands = []
+        self.finished_changes = []
+
+    def getProcessOutput(self, args):
+        d = defer.Deferred()
+        self.pending_commands.append((args, d))
+        return d
+
+    def submit_changes(self, changes):
+        self.finished_changes.extend(changes)
+
 class ComputeChanges(unittest.TestCase):
     def test1(self):
         base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample"
         s = SvnSource(base)
         s._prefix = "sample"
-        doc = s._parse_logs(changes_output_1)
+        output = make_changes_output(4)
+        doc = s.parse_logs(output)
 
-        newlast, logentries = s._filter_new_logentries(doc, 3)
-        self.failUnlessEqual(newlast, 3)
+        newlast, logentries = s._filter_new_logentries(doc, 4)
+        self.failUnlessEqual(newlast, 4)
         self.failUnlessEqual(len(logentries), 0)
 
-        newlast, logentries = s._filter_new_logentries(doc, 2)
-        self.failUnlessEqual(newlast, 3)
+        newlast, logentries = s._filter_new_logentries(doc, 3)
+        self.failUnlessEqual(newlast, 4)
         self.failUnlessEqual(len(logentries), 1)
 
-        newlast, logentries = s._filter_new_logentries(doc, 0)
-        self.failUnlessEqual(newlast, 3)
-        self.failUnlessEqual(len(logentries), 2)
+        newlast, logentries = s._filter_new_logentries(doc, 1)
+        self.failUnlessEqual(newlast, 4)
+        self.failUnlessEqual(len(logentries), 3)
 
-    def split_file(self, path):
-        pieces = path.split("/")
-        if pieces[0] == "branch":
-            return "branch", "/".join(pieces[1:])
-        if pieces[0] == "trunk":
-            return None, "/".join(pieces[1:])
-        raise RuntimeError("there shouldn't be any files like %s" % path)
+        newlast, logentries = s._filter_new_logentries(doc, None)
+        self.failUnlessEqual(newlast, 4)
+        self.failUnlessEqual(len(logentries), 0)
 
     def testChanges(self):
         base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample"
-        s = SvnSource(base, split_file=self.split_file)
+        s = SvnSource(base, split_file=split_file)
         s._prefix = "sample"
-        doc = s._parse_logs(changes_output_1)
-        newlast, logentries = s._filter_new_logentries(doc, 0)
-        changes = s._create_changes(logentries)
+        doc = s.parse_logs(make_changes_output(3))
+        newlast, logentries = s._filter_new_logentries(doc, 1)
+        # so we see revisions 2 and 3 as being new
+        self.failUnlessEqual(newlast, 3)
+        changes = s.create_changes(logentries)
         self.failUnlessEqual(len(changes), 2)
         self.failUnlessEqual(changes[0].branch, "branch")
+        self.failUnlessEqual(changes[0].revision, 2)
         self.failUnlessEqual(changes[1].branch, "branch")
         self.failUnlessEqual(changes[1].files, ["main.c"])
+        self.failUnlessEqual(changes[1].revision, 3)
+
+        # and now pull in r4
+        doc = s.parse_logs(make_changes_output(4))
+        newlast, logentries = s._filter_new_logentries(doc, newlast)
+        self.failUnlessEqual(newlast, 4)
+        # so we see revision 4 as being new
+        changes = s.create_changes(logentries)
+        self.failUnlessEqual(len(changes), 1)
+        self.failUnlessEqual(changes[0].branch, None)
+        self.failUnlessEqual(changes[0].revision, 4)
+        self.failUnlessEqual(changes[0].files, ["version.c"])
+
+    def testFirstTime(self):
+        base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample"
+        s = SvnSource(base, split_file=split_file)
+        s._prefix = "sample"
+        doc = s.parse_logs(make_changes_output(4))
+        logentries = s.get_new_logentries(doc)
+        # SvnSource ignores all changes that happened before it was started
+        self.failUnlessEqual(len(logentries), 0)
+        self.failUnlessEqual(s.last_change, 4)
+
+class Misc(unittest.TestCase):
+    def testAlreadyWorking(self):
+        base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample"
+        s = MySvnSource(base)
+        d = s.checksvn()
+        # the SvnSource is now waiting for its getProcessOutput to finish
+        self.failUnlessEqual(s.overrun_counter, 0)
+        d2 = s.checksvn()
+        self.failUnlessEqual(s.overrun_counter, 1)
+        self.failUnlessEqual(len(s.pending_commands), 1)
+
+    def testGetRoot(self):
+        base = "svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk"
+        s = MySvnSource(base)
+        d = s.checksvn()
+        # the SvnSource is now waiting for its getProcessOutput to finish
+        self.failUnlessEqual(len(s.pending_commands), 1)
+        self.failUnlessEqual(s.pending_commands[0][0],
+                             ["info", "--xml", "--non-interactive", base])
+
+def makeTime(timestring):
+    datefmt = '%Y/%m/%d %H:%M:%S'
+    when = time.mktime(time.strptime(timestring, datefmt))
+    return when
+
+
+class Everything(unittest.TestCase):
+    def test1(self):
+        s = MySvnSource(sample_base, split_file=split_file)
+        d = s.checksvn()
+        # the SvnSource is now waiting for its getProcessOutput to finish
+        self.failUnlessEqual(len(s.pending_commands), 1)
+        self.failUnlessEqual(s.pending_commands[0][0],
+                             ["info", "--xml", "--non-interactive",
+                              sample_base])
+        d = s.pending_commands[0][1]
+        s.pending_commands.pop(0)
+        d.callback(sample_info_output)
+        # now it should be waiting for the 'svn log' command
+        self.failUnlessEqual(len(s.pending_commands), 1)
+        self.failUnlessEqual(s.pending_commands[0][0],
+                             ["log", "--xml", "--verbose", "--non-interactive",
+                              "--limit=100", sample_base])
+        d = s.pending_commands[0][1]
+        s.pending_commands.pop(0)
+        d.callback(make_changes_output(1))
+        # the command ignores the first batch of changes
+        self.failUnlessEqual(len(s.finished_changes), 0)
+        self.failUnlessEqual(s.last_change, 1)
+
+        # now fire it again, nothing changing
+        d = s.checksvn()
+        self.failUnlessEqual(s.pending_commands[0][0],
+                             ["log", "--xml", "--verbose", "--non-interactive",
+                              "--limit=100", sample_base])
+        d = s.pending_commands[0][1]
+        s.pending_commands.pop(0)
+        d.callback(make_changes_output(1))
+        # nothing has changed
+        self.failUnlessEqual(len(s.finished_changes), 0)
+        self.failUnlessEqual(s.last_change, 1)
+
+        # and again, with r2 this time
+        d = s.checksvn()
+        self.failUnlessEqual(s.pending_commands[0][0],
+                             ["log", "--xml", "--verbose", "--non-interactive",
+                              "--limit=100", sample_base])
+        d = s.pending_commands[0][1]
+        s.pending_commands.pop(0)
+        d.callback(make_changes_output(2))
+        # r2 should appear
+        self.failUnlessEqual(len(s.finished_changes), 1)
+        self.failUnlessEqual(s.last_change, 2)
+
+        c = s.finished_changes[0]
+        self.failUnlessEqual(c.branch, "branch")
+        self.failUnlessEqual(c.revision, 2)
+        self.failUnlessEqual(c.files, [''])
+        # TODO: this is what creating the branch looks like: a Change with a
+        # zero-length file. We should decide if we want filenames like this
+        # in the Change (and make sure nobody else gets confused by it) or if
+        # we want to strip them out.
+        self.failUnlessEqual(c.comments, "make_branch")
+
+        # and again at r2, so nothing should change
+        d = s.checksvn()
+        self.failUnlessEqual(s.pending_commands[0][0],
+                             ["log", "--xml", "--verbose", "--non-interactive",
+                              "--limit=100", sample_base])
+        d = s.pending_commands[0][1]
+        s.pending_commands.pop(0)
+        d.callback(make_changes_output(2))
+        # nothing has changed
+        self.failUnlessEqual(len(s.finished_changes), 1)
+        self.failUnlessEqual(s.last_change, 2)
+
+        # and again with both r3 and r4 appearing together
+        d = s.checksvn()
+        self.failUnlessEqual(s.pending_commands[0][0],
+                             ["log", "--xml", "--verbose", "--non-interactive",
+                              "--limit=100", sample_base])
+        d = s.pending_commands[0][1]
+        s.pending_commands.pop(0)
+        d.callback(make_changes_output(4))
+        self.failUnlessEqual(len(s.finished_changes), 3)
+        self.failUnlessEqual(s.last_change, 4)
+
+        c3 = s.finished_changes[1]
+        self.failUnlessEqual(c3.branch, "branch")
+        self.failUnlessEqual(c3.revision, 3)
+        self.failUnlessEqual(c3.files, ["main.c"])
+        self.failUnlessEqual(c3.comments, "commit_on_branch")
+
+        c4 = s.finished_changes[2]
+        self.failUnlessEqual(c4.branch, None)
+        self.failUnlessEqual(c4.revision, 4)
+        self.failUnlessEqual(c4.files, ["version.c"])
+        self.failUnlessEqual(c4.comments, "revised_to_2")
+        self.failUnlessEqual(c4.when, makeTime("2006/10/01 19:35:16"))
+
+
+# TODO:
+#  get coverage of split_file returning None
+#  point at a live SVN server for a little while




From warner at users.sourceforge.net  Mon Oct  2 00:13:28 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:28 +0000
Subject: [Buildbot-commits] buildbot/buildbot/test test_svnpoller.py, 1.2,
	1.3
Message-ID: <E1GUBR2-0000L0-Lu@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/test
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17160/buildbot/test

Modified Files:
	test_svnpoller.py 
Log Message:
[project @ svnpoller: clean up timestamps, unicode usage]

Original author: warner at lothar.com
Date: 2006-10-02 00:03:58

Index: test_svnpoller.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_svnpoller.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -u -d -r1.2 -r1.3
--- test_svnpoller.py	2 Oct 2006 00:13:18 -0000	1.2
+++ test_svnpoller.py	2 Oct 2006 00:13:26 -0000	1.3
@@ -421,7 +421,7 @@
         self.failUnlessEqual(c4.revision, 4)
         self.failUnlessEqual(c4.files, ["version.c"])
         self.failUnlessEqual(c4.comments, "revised_to_2")
-        self.failUnlessEqual(c4.when, makeTime("2006/10/01 19:35:16"))
+        self.failUnless(abs(c4.when - time.time()) < 60)
 
 
 # TODO:




From warner at users.sourceforge.net  Mon Oct  2 00:13:28 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:28 +0000
Subject: [Buildbot-commits] buildbot ChangeLog.svnpoller,1.1,1.2
Message-ID: <E1GUBR2-0005EQ-JA@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17160

Modified Files:
	ChangeLog.svnpoller 
Log Message:
[project @ svnpoller: clean up timestamps, unicode usage]

Original author: warner at lothar.com
Date: 2006-10-02 00:03:58

Index: ChangeLog.svnpoller
===================================================================
RCS file: /cvsroot/buildbot/buildbot/ChangeLog.svnpoller,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -d -r1.1 -r1.2
--- ChangeLog.svnpoller	2 Oct 2006 00:13:11 -0000	1.1
+++ ChangeLog.svnpoller	2 Oct 2006 00:13:26 -0000	1.2
@@ -1,3 +1,18 @@
+Sun Oct  1 16:58:28 2006  Brian Warner  <warner at lothar.com>
+
+	* buildbot/changes/svnpoller.py (SvnSource.create_changes):
+	de-unicodify filenames before creating the Change, because the
+	rest of buildbot is unlikely to handle them well. Leave the 'who'
+	field as a unicode object.. I don't think there's anything that
+	will break very soon, and it will probably nudge us towards
+	accepting unicode everywhere sooner or later. Stop using the
+	"date" field that comes out of SVN, since it is using the
+	repository's clock (and timezone) and what we care about is the
+	buildmaster's (otherwise Changes from the future show up later
+	than the builds they triggered).
+	* buildbot/test/test_svnpoller.py (Everything.test1): match the
+	change to .when
+
 Sun Oct  1 10:12:44 2006  Brian Warner  <warner at lothar.com>
 
 	* buildbot/changes/svnpoller.py (SvnSource): added Niklaus Giger's




From warner at users.sourceforge.net  Mon Oct  2 00:13:28 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:28 +0000
Subject: [Buildbot-commits] buildbot/buildbot/changes svnpoller.py,1.2,1.3
Message-ID: <E1GUBR2-0005EV-KZ@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/changes
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17160/buildbot/changes

Modified Files:
	svnpoller.py 
Log Message:
[project @ svnpoller: clean up timestamps, unicode usage]

Original author: warner at lothar.com
Date: 2006-10-02 00:03:58

Index: svnpoller.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/changes/svnpoller.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -u -d -r1.2 -r1.3
--- svnpoller.py	2 Oct 2006 00:13:18 -0000	1.2
+++ svnpoller.py	2 Oct 2006 00:13:26 -0000	1.3
@@ -392,15 +392,27 @@
             # of that string up to b.s.source.SVN methods
             revision = int(el.getAttribute("revision"))
             dbgMsg("Adding change revision %s" % (revision,))
+            # TODO: the rest of buildbot may not be ready for unicode 'who'
+            # values
             author   = self._get_text(el, "author")
             comments = self._get_text(el, "msg")
-            when     = self._get_text(el, "date")
-            when     = time.mktime(time.strptime("%.19s" % when,
-                                                 "%Y-%m-%dT%H:%M:%S"))
+            # there is a "date" field, but it provides localtime in the
+            # repository's timezone, whereas we care about buildmaster's
+            # localtime (since this will get used to position the boxes on
+            # the Waterfall display, etc). So ignore the date field and use
+            # our local clock instead.
+            #when     = self._get_text(el, "date")
+            #when     = time.mktime(time.strptime("%.19s" % when,
+            #                                     "%Y-%m-%dT%H:%M:%S"))
             branches = {}
             pathlist = el.getElementsByTagName("paths")[0]
             for p in pathlist.getElementsByTagName("path"):
                 path = "".join([t.data for t in p.childNodes])
+                # the rest of buildbot is certaily not yet ready to handle
+                # unicode filenames, because they get put in RemoteCommands
+                # which get sent via PB to the buildslave, and PB doesn't
+                # handle unicode.
+                path = path.encode("ascii")
                 if path.startswith("/"):
                     path = path[1:]
                 where = self._transform_path(path)
@@ -417,7 +429,6 @@
                            files=branches[branch],
                            comments=comments,
                            revision=revision,
-                           when=when,
                            branch=branch)
                 changes.append(c)
 




From warner at users.sourceforge.net  Mon Oct  2 00:13:34 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:34 +0000
Subject: [Buildbot-commits] buildbot ChangeLog.svnpoller,1.2,1.3
Message-ID: <E1GUBR8-0000MH-PY@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17183

Modified Files:
	ChangeLog.svnpoller 
Log Message:
[project @ html.py: improve unicode handling from zero to 'enough']

Original author: warner at lothar.com
Date: 2006-10-02 00:04:42

Index: ChangeLog.svnpoller
===================================================================
RCS file: /cvsroot/buildbot/buildbot/ChangeLog.svnpoller,v
retrieving revision 1.2
retrieving revision 1.3
diff -u -d -r1.2 -r1.3
--- ChangeLog.svnpoller	2 Oct 2006 00:13:26 -0000	1.2
+++ ChangeLog.svnpoller	2 Oct 2006 00:13:32 -0000	1.3
@@ -1,5 +1,10 @@
 Sun Oct  1 16:58:28 2006  Brian Warner  <warner at lothar.com>
 
+	* buildbot/status/html.py (HtmlResource.render): if we get a
+	unicode object from our content() method, encode it into utf-8
+	like we've been claiming to all along. This allows the comments
+	and author names from svnpoller.py to be delivered properly.
+
 	* buildbot/changes/svnpoller.py (SvnSource.create_changes):
 	de-unicodify filenames before creating the Change, because the
 	rest of buildbot is unlikely to handle them well. Leave the 'who'




From warner at users.sourceforge.net  Mon Oct  2 00:13:34 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:34 +0000
Subject: [Buildbot-commits] buildbot/buildbot/status html.py,1.91,1.92
Message-ID: <E1GUBR8-0005GD-Q2@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/status
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17183/buildbot/status

Modified Files:
	html.py 
Log Message:
[project @ html.py: improve unicode handling from zero to 'enough']

Original author: warner at lothar.com
Date: 2006-10-02 00:04:42

Index: html.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/status/html.py,v
retrieving revision 1.91
retrieving revision 1.92
diff -u -d -r1.91 -r1.92
--- html.py	25 Sep 2006 02:43:56 -0000	1.91
+++ html.py	2 Oct 2006 00:13:32 -0000	1.92
@@ -144,14 +144,18 @@
 class HtmlResource(Resource):
     css = None
     contentType = "text/html; charset=UTF-8"
+    title = "Dummy"
+
     def render(self, request):
         data = self.content(request)
+        if isinstance(data, unicode):
+            data = data.encode("utf-8")
         request.setHeader("content-type", self.contentType)
         if request.method == "HEAD":
             request.setHeader("content-length", len(data))
             return ''
         return data
-    title = "Dummy"
+
     def content(self, request):
         data = ('<!DOCTYPE html PUBLIC'
                 ' "-//W3C//DTD XHTML 1.0 Transitional//EN"\n'
@@ -172,6 +176,7 @@
         data += self.body(request)
         data += "</body></html>\n"
         return data
+
     def body(self, request):
         return "Dummy\n"
 




From warner at users.sourceforge.net  Mon Oct  2 00:13:42 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Mon, 02 Oct 2006 00:13:42 +0000
Subject: [Buildbot-commits] buildbot ChangeLog, 1.761,
	1.762 ChangeLog.svnpoller, 1.3, NONE
Message-ID: <E1GUBRH-0000O4-2Z@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17209

Modified Files:
	ChangeLog 
Removed Files:
	ChangeLog.svnpoller 
Log Message:
[project @ svnpoller: merge ChangeLog entries]

Original author: warner at lothar.com
Date: 2006-10-02 00:08:59

Index: ChangeLog
===================================================================
RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v
retrieving revision 1.761
retrieving revision 1.762
diff -u -d -r1.761 -r1.762
--- ChangeLog	30 Sep 2006 22:36:22 -0000	1.761
+++ ChangeLog	2 Oct 2006 00:13:40 -0000	1.762
@@ -1,3 +1,30 @@
+2006-10-01  Brian Warner  <warner at lothar.com>
+
+	* buildbot/status/html.py (HtmlResource.render): if we get a
+	unicode object from our content() method, encode it into utf-8
+	like we've been claiming to all along. This allows the comments
+	and author names from svnpoller.py to be delivered properly.
+
+	* buildbot/changes/svnpoller.py (SvnSource.create_changes):
+	de-unicodify filenames before creating the Change, because the
+	rest of buildbot is unlikely to handle them well. Leave the 'who'
+	field as a unicode object.. I don't think there's anything that
+	will break very soon, and it will probably nudge us towards
+	accepting unicode everywhere sooner or later. Stop using the
+	"date" field that comes out of SVN, since it is using the
+	repository's clock (and timezone) and what we care about is the
+	buildmaster's (otherwise Changes from the future show up later
+	than the builds they triggered).
+	* buildbot/test/test_svnpoller.py (Everything.test1): match the
+	change to .when
+
+	* buildbot/changes/svnpoller.py (SvnSource): added Niklaus Giger's
+	Suvbersion repository polling ChangeSource. I've hacked it up
+	considerably: any bugs are entirely my own fault. Thank you
+	Niklaus!
+	* buildbot/test/test_svnpoller.py: tests for it
+	* docs/buildbot.texinfo (SvnSource): document it
+
 2006-09-30  Brian Warner  <warner at lothar.com>
 
 	* buildbot/scheduler.py (Periodic): submit a reason= to the

--- ChangeLog.svnpoller DELETED ---




From warner at users.sourceforge.net  Tue Oct  3 17:23:16 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Tue, 03 Oct 2006 17:23:16 +0000
Subject: [Buildbot-commits] buildbot setup.py,1.40,1.41
Message-ID: <E1GUnzB-0000hv-7O@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv22148

Modified Files:
	setup.py 
Log Message:
[project @ setup.py: fix build/install errors involving unit test data files]

Original author: warner at lothar.com
Date: 2006-10-03 17:13:27

Index: setup.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/setup.py,v
retrieving revision 1.40
retrieving revision 1.41
diff -u -d -r1.40 -r1.41
--- setup.py	25 Sep 2006 08:12:22 -0000	1.40
+++ setup.py	3 Oct 2006 17:23:14 -0000	1.41
@@ -41,7 +41,8 @@
 for f in os.listdir("buildbot/test/mail"):
     if f.endswith("~"):
         continue
-    testmsgs.append("buildbot/test/mail/%s" % f)
+    if f.startswith("msg") or f.startswith("syncmail"):
+        testmsgs.append("buildbot/test/mail/%s" % f)
 
 setup(name="buildbot",
       version=version,
@@ -71,13 +72,13 @@
                 "buildbot.slave",
                 "buildbot.scripts",
                 "buildbot.test",
-                "buildbot.test.subdir",
                 ],
       data_files=[("buildbot", ["buildbot/buildbot.png"]),
                   ("buildbot/clients", ["buildbot/clients/debug.glade"]),
                   ("buildbot/status", ["buildbot/status/classic.css"]),
                   ("buildbot/scripts", ["buildbot/scripts/sample.cfg"]),
                   ("buildbot/test/mail", testmsgs),
+                  ("buildbot/test/subdir", ["buildbot/test/subdir/emit.py"]),
                   ],
       scripts = scripts,
       cmdclass={'install_data': install_data_twisted},




From warner at users.sourceforge.net  Thu Oct  5 00:45:50 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Thu, 05 Oct 2006 00:45:50 +0000
Subject: [Buildbot-commits] buildbot/buildbot/steps python.py,1.4,1.5
Message-ID: <E1GVHN1-0001AP-K1@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/steps
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv25037/buildbot/steps

Modified Files:
	python.py 
Log Message:
[project @ PyFlakes: ignore initial output lines that weren't emitted by pyflakes]

Original author: warner at allmydata.com
Date: 2006-10-05 00:39:11

Index: python.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/steps/python.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -u -d -r1.4 -r1.5
--- python.py	20 Sep 2006 06:33:35 -0000	1.4
+++ python.py	5 Oct 2006 00:45:48 -0000	1.5
@@ -65,7 +65,19 @@
             counts[m] = 0
             summaries[m] = []
 
+        first = True
         for line in StringIO(log.getText()).readlines():
+            # the first few lines might contain echoed commands from a 'make
+            # pyflakes' step, so don't count these as warnings. Stop ignoring
+            # the initial lines as soon as we see one with a colon.
+            if first:
+                if line.find(":") != -1:
+                    # there's the colon, this is the first real line
+                    first = False
+                    # fall through and parse the line
+                else:
+                    # skip this line, keep skipping non-colon lines
+                    continue
             if line.find("imported but unused") != -1:
                 m = "unused"
             elif line.find("*' used; unable to detect undefined names") != -1:




From warner at users.sourceforge.net  Thu Oct  5 00:45:50 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Thu, 05 Oct 2006 00:45:50 +0000
Subject: [Buildbot-commits] buildbot ChangeLog,1.762,1.763
Message-ID: <E1GVHN0-0008Qs-L3@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv25037

Modified Files:
	ChangeLog 
Log Message:
[project @ PyFlakes: ignore initial output lines that weren't emitted by pyflakes]

Original author: warner at allmydata.com
Date: 2006-10-05 00:39:11

Index: ChangeLog
===================================================================
RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v
retrieving revision 1.762
retrieving revision 1.763
diff -u -d -r1.762 -r1.763
--- ChangeLog	2 Oct 2006 00:13:40 -0000	1.762
+++ ChangeLog	5 Oct 2006 00:45:47 -0000	1.763
@@ -1,3 +1,11 @@
+2006-10-04  Brian Warner  <warner at lothar.com>
+
+	* buildbot/steps/python.py (PyFlakes.createSummary): skip any
+	initial lines that weren't emitted by pyflakes. When the pyflakes
+	command is run under a Makefile, 'make' will echo the command it
+	runs to stdio, and that was getting logged as a "misc" warning.
+	* buildbot/test/test_steps.py (Python.testPyFlakes2): test it
+
 2006-10-01  Brian Warner  <warner at lothar.com>
 
 	* buildbot/status/html.py (HtmlResource.render): if we get a




From warner at users.sourceforge.net  Thu Oct  5 00:45:50 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Thu, 05 Oct 2006 00:45:50 +0000
Subject: [Buildbot-commits] buildbot/buildbot/test test_steps.py,1.31,1.32
Message-ID: <E1GVHN1-0008R1-4C@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/test
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv25037/buildbot/test

Modified Files:
	test_steps.py 
Log Message:
[project @ PyFlakes: ignore initial output lines that weren't emitted by pyflakes]

Original author: warner at allmydata.com
Date: 2006-10-05 00:39:11

Index: test_steps.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_steps.py,v
retrieving revision 1.31
retrieving revision 1.32
diff -u -d -r1.31 -r1.32
--- test_steps.py	19 Sep 2006 18:46:55 -0000	1.31
+++ test_steps.py	5 Oct 2006 00:45:48 -0000	1.32
@@ -394,11 +394,12 @@
         return maybeWait(d)
 
 class Python(StepTester, unittest.TestCase):
-    def testPyFlakes(self):
-        self.masterbase = "Python.master"
+    def testPyFlakes1(self):
+        self.masterbase = "Python.testPyFlakes1"
         step = self.makeStep(python.PyFlakes)
         output = \
-"""buildbot/changes/freshcvsmail.py:5: 'FCMaildirSource' imported but unused
+"""pyflakes buildbot
+buildbot/changes/freshcvsmail.py:5: 'FCMaildirSource' imported but unused
 buildbot/clients/debug.py:9: redefinition of unused 'gtk' from line 9
 buildbot/clients/debug.py:9: 'gnome' imported but unused
 buildbot/scripts/runner.py:323: redefinition of unused 'run' from line 321
@@ -415,6 +416,7 @@
         self.failUnless("undefined=1" in desc)
         self.failUnless("redefs=3" in desc)
         self.failUnless("import*=1" in desc)
+        self.failIf("misc=" in desc)
 
         self.failUnlessEqual(step.getProperty("pyflakes-unused"), 2)
         self.failUnlessEqual(step.getProperty("pyflakes-undefined"), 1)
@@ -438,3 +440,31 @@
         cmd.rc = 0
         results = step.evaluateCommand(cmd)
         self.failUnlessEqual(results, FAILURE) # because of the 'undefined'
+
+    def testPyFlakes2(self):
+        self.masterbase = "Python.testPyFlakes2"
+        step = self.makeStep(python.PyFlakes)
+        output = \
+"""pyflakes buildbot
+some more text here that should be ignored
+buildbot/changes/freshcvsmail.py:5: 'FCMaildirSource' imported but unused
+buildbot/clients/debug.py:9: redefinition of unused 'gtk' from line 9
+buildbot/clients/debug.py:9: 'gnome' imported but unused
+buildbot/scripts/runner.py:323: redefinition of unused 'run' from line 321
+buildbot/scripts/runner.py:325: redefinition of unused 'run' from line 323
+buildbot/scripts/imaginary.py:12: undefined name 'size'
+could not compile 'blah/blah.py':3:
+pretend there was an invalid line here
+buildbot/scripts/imaginary.py:18: 'from buildbot import *' used; unable to detect undefined names
+"""
+        log = step.addLog("stdio")
+        log.addStdout(output)
+        log.finish()
+        step.createSummary(log)
+        desc = step.descriptionDone
+        self.failUnless("unused=2" in desc)
+        self.failUnless("undefined=1" in desc)
+        self.failUnless("redefs=3" in desc)
+        self.failUnless("import*=1" in desc)
+        self.failUnless("misc=2" in desc)
+




From warner at users.sourceforge.net  Thu Oct  5 00:46:00 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Thu, 05 Oct 2006 00:46:00 +0000
Subject: [Buildbot-commits] buildbot ChangeLog,1.763,1.764
Message-ID: <E1GVHNB-0001Da-LY@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv25060

Modified Files:
	ChangeLog 
Log Message:
[project @ PyFlakes: another test]

Original author: warner at allmydata.com
Date: 2006-10-05 00:42:17

Index: ChangeLog
===================================================================
RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v
retrieving revision 1.763
retrieving revision 1.764
diff -u -d -r1.763 -r1.764
--- ChangeLog	5 Oct 2006 00:45:47 -0000	1.763
+++ ChangeLog	5 Oct 2006 00:45:57 -0000	1.764
@@ -5,6 +5,7 @@
 	command is run under a Makefile, 'make' will echo the command it
 	runs to stdio, and that was getting logged as a "misc" warning.
 	* buildbot/test/test_steps.py (Python.testPyFlakes2): test it
+	* buildbot/test/test_steps.py (testPyFlakes3): another test
 
 2006-10-01  Brian Warner  <warner at lothar.com>
 




From warner at users.sourceforge.net  Thu Oct  5 00:46:00 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Thu, 05 Oct 2006 00:46:00 +0000
Subject: [Buildbot-commits] buildbot/buildbot/test test_steps.py,1.32,1.33
Message-ID: <E1GVHNB-0001DL-HU@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/test
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv25060/buildbot/test

Modified Files:
	test_steps.py 
Log Message:
[project @ PyFlakes: another test]

Original author: warner at allmydata.com
Date: 2006-10-05 00:42:17

Index: test_steps.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_steps.py,v
retrieving revision 1.32
retrieving revision 1.33
diff -u -d -r1.32 -r1.33
--- test_steps.py	5 Oct 2006 00:45:48 -0000	1.32
+++ test_steps.py	5 Oct 2006 00:45:58 -0000	1.33
@@ -468,3 +468,27 @@
         self.failUnless("import*=1" in desc)
         self.failUnless("misc=2" in desc)
 
+
+    def testPyFlakes3(self):
+        self.masterbase = "Python.testPyFlakes3"
+        step = self.makeStep(python.PyFlakes)
+        output = \
+"""buildbot/changes/freshcvsmail.py:5: 'FCMaildirSource' imported but unused
+buildbot/clients/debug.py:9: redefinition of unused 'gtk' from line 9
+buildbot/clients/debug.py:9: 'gnome' imported but unused
+buildbot/scripts/runner.py:323: redefinition of unused 'run' from line 321
+buildbot/scripts/runner.py:325: redefinition of unused 'run' from line 323
+buildbot/scripts/imaginary.py:12: undefined name 'size'
+buildbot/scripts/imaginary.py:18: 'from buildbot import *' used; unable to detect undefined names
+"""
+        log = step.addLog("stdio")
+        log.addStdout(output)
+        log.finish()
+        step.createSummary(log)
+        desc = step.descriptionDone
+        self.failUnless("unused=2" in desc)
+        self.failUnless("undefined=1" in desc)
+        self.failUnless("redefs=3" in desc)
+        self.failUnless("import*=1" in desc)
+        self.failIf("misc" in desc)
+




From warner at users.sourceforge.net  Sun Oct  8 20:54:19 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Sun, 08 Oct 2006 20:54:19 +0000
Subject: [Buildbot-commits] buildbot/contrib svn_watcher.py,1.1,1.2
Message-ID: <E1GWffA-0006Q5-0p@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/contrib
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6339/contrib

Modified Files:
	svn_watcher.py 
Log Message:
[project @ security fix to contrib/svn_watcher.py]

Original author: warner at lothar.com
Date: 2006-10-08 20:48:35

Index: svn_watcher.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/contrib/svn_watcher.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -d -r1.1 -r1.2
--- svn_watcher.py	17 Apr 2006 18:11:42 -0000	1.1
+++ svn_watcher.py	8 Oct 2006 20:54:17 -0000	1.2
@@ -16,7 +16,7 @@
 
 # 15.03.06 by John Pye
 # 29.03.06 by Niklaus Giger, added support to run under windows, added invocation option
-import commands
+import subprocess
 import xml.dom.minidom
 import sys
 import time
@@ -24,8 +24,13 @@
 if sys.platform == 'win32':
     import win32pipe
 
+def getoutput(cmd):
+    p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
+    return p.stdout.read()
+
 def checkChanges(repo, master, verbose=False, oldRevision=-1):
-    cmd ="svn log --non-interactive --xml --verbose --limit=1 "+repo
+    cmd = ["svn", "log", "--non-interactive", "--xml", "--verbose",
+           "--limit=1", repo]
     if verbose == True:
         print "Getting last revision of repository: " + repo
 
@@ -34,11 +39,11 @@
         xml1 = ''.join(f.readlines())
         f.close()
     else:
-        xml1 = commands.getoutput(cmd)  
-    
+        xml1 = getoutput(cmd)
+
     if verbose == True:
         print "XML\n-----------\n"+xml1+"\n\n"
-    
+
     doc = xml.dom.minidom.parseString(xml1)
     el = doc.getElementsByTagName("logentry")[0]
     revision = el.getAttribute("revision")
@@ -57,20 +62,23 @@
         print paths
 
     if  revision != oldRevision:
-        cmd = "buildbot sendchange --master="+master+" --revision=\""+revision+"\" --username=\""+author+"\"--comments=\""+comments+"\" "+" ".join(paths)
-    
+        cmd = ["buildbot", "sendchange", "--master=%s"%master,
+               "--revision=%s"%revision, "--username=%s"%author,
+               "--comments=%s"%comments]
+        cmd += paths
+
         if verbose == True:
             print cmd
-    
+
         if sys.platform == 'win32':
             f = win32pipe.popen(cmd)
             print time.strftime("%H.%M.%S ") + "Revision "+revision+ ": "+ ''.join(f.readlines())
             f.close()
         else:
-            xml1 = commands.getoutput(cmd)  
+            xml1 = getoutput(cmd)
     else:
         print time.strftime("%H.%M.%S ") + "nothing has changed since revision "+revision
-       
+
     return revision
 
 if __name__ == '__main__':




From warner at users.sourceforge.net  Sun Oct  8 20:54:19 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Sun, 08 Oct 2006 20:54:19 +0000
Subject: [Buildbot-commits] buildbot CREDITS,1.4,1.5 ChangeLog,1.764,1.765
Message-ID: <E1GWffA-0006Q6-0h@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6339

Modified Files:
	CREDITS ChangeLog 
Log Message:
[project @ security fix to contrib/svn_watcher.py]

Original author: warner at lothar.com
Date: 2006-10-08 20:48:35

Index: CREDITS
===================================================================
RCS file: /cvsroot/buildbot/buildbot/CREDITS,v
retrieving revision 1.4
retrieving revision 1.5
diff -u -d -r1.4 -r1.5
--- CREDITS	17 Sep 2006 20:49:31 -0000	1.4
+++ CREDITS	8 Oct 2006 20:54:16 -0000	1.5
@@ -44,3 +44,4 @@
 Albert Hofkamp
 Brett Neely
 Wade Brainerd
+Nick Mathewson

Index: ChangeLog
===================================================================
RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v
retrieving revision 1.764
retrieving revision 1.765
diff -u -d -r1.764 -r1.765
--- ChangeLog	5 Oct 2006 00:45:57 -0000	1.764
+++ ChangeLog	8 Oct 2006 20:54:16 -0000	1.765
@@ -1,3 +1,12 @@
+2006-10-08  Brian Warner  <warner at lothar.com>
+
+	* contrib/svn_watcher.py: fix security holes by using proper argv
+	arrays and subprocess.Popen() rather than commands.getoutput().
+	Thanks to Nick Mathewson for the patch. Note that svn_watcher.py
+	is deprecated in favor of buildbot/changes/svnpoller.py, and will
+	probably be removed by the next release.
+	* CREDITS: add Nick
+
 2006-10-04  Brian Warner  <warner at lothar.com>
 
 	* buildbot/steps/python.py (PyFlakes.createSummary): skip any




From warner at users.sourceforge.net  Fri Oct 13 07:30:03 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Fri, 13 Oct 2006 07:30:03 +0000
Subject: [Buildbot-commits] buildbot/buildbot/steps transfer.py,1.3,1.4
Message-ID: <E1GYHUb-0007q9-2N@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/steps
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6047/buildbot/steps

Modified Files:
	transfer.py 
Log Message:
[project @ filetransfer: code cleanups]

Original author: warner at lothar.com
Date: 2006-10-13 07:28:48

Index: transfer.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/steps/transfer.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -u -d -r1.3 -r1.4
--- transfer.py	17 Sep 2006 20:35:49 -0000	1.3
+++ transfer.py	13 Oct 2006 07:29:58 -0000	1.4
@@ -104,20 +104,20 @@
 
     name = 'upload'
 
-    def __init__(self, build, **kwargs):
-        buildstep_kwargs = {}
-        for k in kwargs.keys()[:]:
-            if k in BuildStep.parms:
-                buildstep_kwargs[k] = kwargs[k]
-                del kwargs[k]
-            BuildStep.__init__(self, build, **buildstep_kwargs)
+    def __init__(self, build, slavesrc, masterdest,
+                 workdir="build", maxsize=None, blocksize=16*1024,
+                 **buildstep_kwargs):
+        BuildStep.__init__(self, build, **buildstep_kwargs)
 
-        self.args = kwargs
-        self.fileWriter = None
+        self.slavesrc = slavesrc
+        self.masterdest = masterdest
+        self.workdir = workdir
+        self.maxsize = maxsize
+        self.blocksize = blocksize
 
     def start(self):
-        source = self.args['slavesrc']
-        masterdest = self.args['masterdest']
+        source = self.slavesrc
+        masterdest = self.masterdest
         # we rely upon the fact that the buildmaster runs chdir'ed into its
         # basedir to make sure that relative paths in masterdest are expanded
         # properly. TODO: maybe pass the master's basedir all the way down
@@ -134,12 +134,12 @@
 
         # default arguments
         args = {
-            'maxsize': None,
-            'blocksize': 16*1024,
-            'workdir': 'build',
+            'slavesrc': source,
+            'workdir': self.workdir,
+            'writer': self.fileWriter,
+            'maxsize': self.maxsize,
+            'blocksize': self.blocksize
             }
-        args.update(self.args)
-        args['writer'] = self.fileWriter
 
         self.cmd = StatusRemoteCommand('uploadFile', args)
         d = self.runCommand(self.cmd)
@@ -173,20 +173,20 @@
 
     name = 'download'
 
-    def __init__(self,build, **kwargs):
-        buildstep_kwargs = {}
-        for k in kwargs.keys()[:]:
-            if k in BuildStep.parms:
-                buildstep_kwargs[k] = kwargs[k]
-                del kwargs[k]
-            BuildStep.__init__(self, build, **buildstep_kwargs)
+    def __init__(self, build, mastersrc, slavedest,
+                 workdir="build", maxsize=None, blocksize=16*1024,
+                 **buildstep_kwargs):
+        BuildStep.__init__(self, build, **buildstep_kwargs)
 
-        self.args = kwargs
-        self.fileReader = None
+        self.mastersrc = mastersrc
+        self.slavedest = slavedest
+        self.workdir = workdir
+        self.maxsize = maxsize
+        self.blocksize = blocksize
 
     def start(self):
-        source = os.path.expanduser(self.args['mastersrc'])
-        slavedest = self.args['slavedest']
+        source = os.path.expanduser(self.mastersrc)
+        slavedest = self.slavedest
         log.msg("FileDownload started, from master %r to slave %r" %
                 (source, slavedest))
 
@@ -206,12 +206,12 @@
 
         # default arguments
         args = {
-            'maxsize': None,
-            'blocksize': 16*1024,
-            'workdir': 'build',
+            'slavedest': self.slavedest,
+            'maxsize': self.maxsize,
+            'reader': self.fileReader,
+            'blocksize': self.blocksize,
+            'workdir': self.workdir,
             }
-        args.update(self.args)
-        args['reader'] = self.fileReader
 
         self.cmd = StatusRemoteCommand('downloadFile', args)
         d = self.runCommand(self.cmd)




From warner at users.sourceforge.net  Fri Oct 13 07:30:00 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Fri, 13 Oct 2006 07:30:00 +0000
Subject: [Buildbot-commits] buildbot ChangeLog,1.765,1.766
Message-ID: <E1GYHV2-0002Jj-VK@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6047

Modified Files:
	ChangeLog 
Log Message:
[project @ filetransfer: code cleanups]

Original author: warner at lothar.com
Date: 2006-10-13 07:28:48

Index: ChangeLog
===================================================================
RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v
retrieving revision 1.765
retrieving revision 1.766
diff -u -d -r1.765 -r1.766
--- ChangeLog	8 Oct 2006 20:54:16 -0000	1.765
+++ ChangeLog	13 Oct 2006 07:29:57 -0000	1.766
@@ -1,3 +1,12 @@
+2006-10-12  Brian Warner  <warner at lothar.com>
+
+	* buildbot/steps/transfer.py: rework __init__ and args setup
+	* buildbot/slave/commands.py (SlaveFileDownloadCommand): minor
+	docs improvements
+	* buildbot/slave/commands.py (SlaveFileDownloadCommand.setup):
+	when opening the target file, only catch IOError (to report via
+	stderr/rc!=0), let the others be reported as normal exceptions
+
 2006-10-08  Brian Warner  <warner at lothar.com>
 
 	* contrib/svn_watcher.py: fix security holes by using proper argv




From warner at users.sourceforge.net  Fri Oct 13 07:30:00 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Fri, 13 Oct 2006 07:30:00 +0000
Subject: [Buildbot-commits] buildbot/buildbot/slave commands.py,1.68,1.69
Message-ID: <E1GYHV5-0002LN-5V@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/slave
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6047/buildbot/slave

Modified Files:
	commands.py 
Log Message:
[project @ filetransfer: code cleanups]

Original author: warner at lothar.com
Date: 2006-10-13 07:28:48

Index: commands.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/slave/commands.py,v
retrieving revision 1.68
retrieving revision 1.69
diff -u -d -r1.68 -r1.69
--- commands.py	30 Sep 2006 19:32:48 -0000	1.68
+++ commands.py	13 Oct 2006 07:29:58 -0000	1.69
@@ -675,14 +675,12 @@
     Upload a file from slave to build master
     Arguments:
 
-        - ['workdir']:   directory to use
-        - ['slavesrc']:  name of the file to upload to the buildmaster
-        - ['writer']:    object for remote writing
+        - ['workdir']:   base directory to use
+        - ['slavesrc']:  name of the slave-side file to read from
+        - ['writer']:    RemoteReference to a transfer._FileWriter object
         - ['maxsize']:   max size (in bytes) of file to write
-        - ['blocksize']: max size for one data block
-
+        - ['blocksize']: max size for each data block
     """
-
     debug = False
 
     def setup(self,args):
@@ -786,12 +784,11 @@
     Download a file from master to slave
     Arguments:
 
-        - ['workdir']:   directory to use
-        - ['slavedest']: name of the file to upload to the buildmaster
-        - ['reader']:    object for remote reading
+        - ['workdir']:   base directory to use
+        - ['slavedest']: name of the slave-side file to be created
+        - ['reader']:    RemoteReference to a transfer._FileReader object
         - ['maxsize']:   max size (in bytes) of file to write
-        - ['blocksize']: max size for one data block
-
+        - ['blocksize']: max size for each data block
     """
     debug = False
 
@@ -815,7 +812,7 @@
             self.fp = open(self.path, 'w')
             if self.debug:
                 log.msg('Opened %r for download' % self.path)
-        except:
+        except IOError:
             self.fp = None
             self.stderr = 'Cannot open file %r for download' % self.path
             self.rc = 1




From warner at users.sourceforge.net  Fri Oct 13 09:08:27 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Fri, 13 Oct 2006 09:08:27 +0000
Subject: [Buildbot-commits] buildbot/buildbot/steps transfer.py,1.4,1.5
Message-ID: <E1GYJ1o-0003sN-3W@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/steps
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv12999/buildbot/steps

Modified Files:
	transfer.py 
Log Message:
[project @ transfer.py: fix a stupid error]

Original author: warner at lothar.com
Date: 2006-10-13 09:06:53

Index: transfer.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/steps/transfer.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -u -d -r1.4 -r1.5
--- transfer.py	13 Oct 2006 07:29:58 -0000	1.4
+++ transfer.py	13 Oct 2006 09:08:25 -0000	1.5
@@ -129,7 +129,7 @@
         self.step_status.setColor('yellow')
         self.step_status.setText(['uploading', source])
 
-        fp = open(self.args['masterdest'],'w')
+        fp = open(self.masterdest, 'w')
         self.fileWriter = _FileWriter(fp)
 
         # default arguments




From warner at users.sourceforge.net  Fri Oct 13 09:08:27 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Fri, 13 Oct 2006 09:08:27 +0000
Subject: [Buildbot-commits] buildbot/buildbot/slave bot.py,1.20,1.21
Message-ID: <E1GYJ1o-0001CL-0m@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot/slave
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv12999/buildbot/slave

Modified Files:
	bot.py 
Log Message:
[project @ transfer.py: fix a stupid error]

Original author: warner at lothar.com
Date: 2006-10-13 09:06:53

Index: bot.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/slave/bot.py,v
retrieving revision 1.20
retrieving revision 1.21
diff -u -d -r1.20 -r1.21
--- bot.py	6 Sep 2006 00:41:55 -0000	1.20
+++ bot.py	13 Oct 2006 09:08:25 -0000	1.21
@@ -65,7 +65,7 @@
         #service.Service.__init__(self) # Service has no __init__ method
         self.setName(name)
         self.not_really = not_really
-        
+
     def __repr__(self):
         return "<SlaveBuilder '%s'>" % self.name
 




From warner at users.sourceforge.net  Fri Oct 13 09:08:27 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Fri, 13 Oct 2006 09:08:27 +0000
Subject: [Buildbot-commits] buildbot ChangeLog,1.766,1.767
Message-ID: <E1GYJ1o-0001CN-0q@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv12999

Modified Files:
	ChangeLog 
Log Message:
[project @ transfer.py: fix a stupid error]

Original author: warner at lothar.com
Date: 2006-10-13 09:06:53

Index: ChangeLog
===================================================================
RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v
retrieving revision 1.766
retrieving revision 1.767
diff -u -d -r1.766 -r1.767
--- ChangeLog	13 Oct 2006 07:29:57 -0000	1.766
+++ ChangeLog	13 Oct 2006 09:08:25 -0000	1.767
@@ -1,3 +1,9 @@
+2006-10-13  Brian Warner  <warner at lothar.com>
+
+	* buildbot/steps/transfer.py (FileUpload.start): Fix stupid error.
+	Maybe I should run my own unit tests before recording a big
+	change. Good thing I've got a buildbot to remind me.
+
 2006-10-12  Brian Warner  <warner at lothar.com>
 
 	* buildbot/steps/transfer.py: rework __init__ and args setup




From warner at users.sourceforge.net  Fri Oct 13 09:14:09 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Fri, 13 Oct 2006 09:14:09 +0000
Subject: [Buildbot-commits] buildbot ChangeLog,1.767,1.768
Message-ID: <E1GYJ7J-0005IF-U5@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv15346

Modified Files:
	ChangeLog 
Log Message:
[project @ docs: update some examples to match reorganized directories]

Original author: warner at lothar.com
Date: 2006-10-13 09:10:06

Index: ChangeLog
===================================================================
RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v
retrieving revision 1.767
retrieving revision 1.768
diff -u -d -r1.767 -r1.768
--- ChangeLog	13 Oct 2006 09:08:25 -0000	1.767
+++ ChangeLog	13 Oct 2006 09:14:07 -0000	1.768
@@ -1,5 +1,8 @@
 2006-10-13  Brian Warner  <warner at lothar.com>
 
+	* docs/buildbot.texinfo (Adding LogObservers): update sample code
+	to match the great Steps renaming
+
 	* buildbot/steps/transfer.py (FileUpload.start): Fix stupid error.
 	Maybe I should run my own unit tests before recording a big
 	change. Good thing I've got a buildbot to remind me.




From warner at users.sourceforge.net  Fri Oct 13 09:14:09 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Fri, 13 Oct 2006 09:14:09 +0000
Subject: [Buildbot-commits] buildbot/docs buildbot.texinfo,1.83,1.84
Message-ID: <E1GYJ7K-0005IP-3T@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/docs
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv15346/docs

Modified Files:
	buildbot.texinfo 
Log Message:
[project @ docs: update some examples to match reorganized directories]

Original author: warner at lothar.com
Date: 2006-10-13 09:10:06

Index: buildbot.texinfo
===================================================================
RCS file: /cvsroot/buildbot/buildbot/docs/buildbot.texinfo,v
retrieving revision 1.83
retrieving revision 1.84
diff -u -d -r1.83 -r1.84
--- buildbot.texinfo	2 Oct 2006 00:13:18 -0000	1.83
+++ buildbot.texinfo	13 Oct 2006 09:14:07 -0000	1.84
@@ -4274,10 +4274,11 @@
 progress that this event represents.
 
 There are a number of pre-built @code{LogObserver} classes that you
-can choose from, and of course you can subclass them to add further
-customization. The @code{LogLineObserver} class handles the grunt work
-of buffering and scanning for end-of-line delimiters, allowing your
-parser to operate on complete stdout/stderr lines.
+can choose from (defined in @code{buildbot.process.buildstep}, and of
+course you can subclass them to add further customization. The
+ at code{LogLineObserver} class handles the grunt work of buffering and
+scanning for end-of-line delimiters, allowing your parser to operate
+on complete stdout/stderr lines.
 
 For example, let's take a look at the @code{TrialTestCaseCounter},
 which is used by the Trial step to count test cases as they are run.
@@ -4298,7 +4299,9 @@
 The parser class looks like this:
 
 @example
-class TrialTestCaseCounter(step.LogLineObserver):
+from buildbot.process.buildstep import LogLineObserver
+
+class TrialTestCaseCounter(LogLineObserver):
     _line_re = re.compile(r'^([\w\.]+) \.\.\. \[([^\]]+)\]$')
     numTests = 0
     finished = False




From warner at users.sourceforge.net  Sun Oct 15 17:51:13 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Sun, 15 Oct 2006 17:51:13 +0000
Subject: [Buildbot-commits] buildbot/buildbot interfaces.py,1.47,1.48
Message-ID: <E1GZA8o-0003zO-AK@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv10284/buildbot

Modified Files:
	interfaces.py 
Log Message:
[project @ IStatus.getURLForThing was misspelled in the interface definition]

Original author: warner at lothar.com
Date: 2006-10-15 17:34:29

Index: interfaces.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/interfaces.py,v
retrieving revision 1.47
retrieving revision 1.48
diff -u -d -r1.47 -r1.48
--- interfaces.py	15 Sep 2006 14:48:53 -0000	1.47
+++ interfaces.py	15 Oct 2006 17:51:11 -0000	1.48
@@ -98,7 +98,7 @@
     def getBuildbotURL():
         """Return the URL of the top-most Buildbot status page, or None if
         this Buildbot does not provide a web status page."""
-    def getURLFor(thing):
+    def getURLForThing(thing):
         """Return the URL of a page which provides information on 'thing',
         which should be an object that implements one of the status
         interfaces defined in L{buildbot.interfaces}. Returns None if no




From warner at users.sourceforge.net  Sun Oct 15 17:51:13 2006
From: warner at users.sourceforge.net (Brian Warner)
Date: Sun, 15 Oct 2006 17:51:13 +0000
Subject: [Buildbot-commits] buildbot CREDITS,1.5,1.6 ChangeLog,1.768,1.769
Message-ID: <E1GZA8o-0003zG-AF@mail.sourceforge.net>

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv10284

Modified Files:
	CREDITS ChangeLog 
Log Message:
[project @ IStatus.getURLForThing was misspelled in the interface definition]

Original author: warner at lothar.com
Date: 2006-10-15 17:34:29

Index: CREDITS
===================================================================
RCS file: /cvsroot/buildbot/buildbot/CREDITS,v
retrieving revision 1.5
retrieving revision 1.6
diff -u -d -r1.5 -r1.6
--- CREDITS	8 Oct 2006 20:54:16 -0000	1.5
+++ CREDITS	15 Oct 2006 17:51:11 -0000	1.6
@@ -45,3 +45,4 @@
 Brett Neely
 Wade Brainerd
 Nick Mathewson
+Roy Rapoport

Index: ChangeLog
===================================================================
RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v
retrieving revision 1.768
retrieving revision 1.769
diff -u -d -r1.768 -r1.769
--- ChangeLog	13 Oct 2006 09:14:07 -0000	1.768
+++ ChangeLog	15 Oct 2006 17:51:11 -0000	1.769
@@ -1,3 +1,9 @@
+2006-10-15  Brian Warner  <warner at lothar.com>
+
+	* buildbot/interfaces.py (IStatus.getURLForThing): oops, the
+	method name was misspelled in the interface definition. Thanks to
+	Roy Rapoport for the catch.
+
 2006-10-13  Brian Warner  <warner at lothar.com>
 
 	* docs/buildbot.texinfo (Adding LogObservers): update sample code