From warner at users.sourceforge.net Mon Oct 2 00:13:13 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:13 +0000 Subject: [Buildbot-commits] buildbot ChangeLog.svnpoller,NONE,1.1 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17108 Added Files: ChangeLog.svnpoller Log Message: [project @ svnpoller: add new ChangeSource, interim checkin] Original author: warner at lothar.com Date: 2006-10-01 17:18:18 --- NEW FILE: ChangeLog.svnpoller --- Sun Oct 1 10:12:44 2006 Brian Warner * buildbot/changes/svnpoller.py (SvnSource): added Niklaus Giger's Suvbersion repository polling ChangeSource. I've hacked it up considerably: any bugs are entirely my own fault. Thank you Niklaus! * buildbot/test/test_svnpoller.py: tests for it * docs/buildbot.texinfo (SvnSource): document it From warner at users.sourceforge.net Mon Oct 2 00:13:13 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:13 +0000 Subject: [Buildbot-commits] buildbot/buildbot/changes svnpoller.py, NONE, 1.1 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/changes In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17108/buildbot/changes Added Files: svnpoller.py Log Message: [project @ svnpoller: add new ChangeSource, interim checkin] Original author: warner at lothar.com Date: 2006-10-01 17:18:18 --- NEW FILE: svnpoller.py --- # -*- test-case-name: buildbot.test.test_svnpoller -*- # Based on the work of Dave Peticolas for the P4poll # Changed to svn (using xml.dom.minidom) by Niklaus Giger import time from twisted.python import log, failure from twisted.internet import defer, reactor from twisted.internet.utils import getProcessOutput from twisted.internet.task import LoopingCall from buildbot import util from buildbot.changes import base from buildbot.changes.changes import Change import xml.dom.minidom def _assert(condition, msg): if condition: return True raise AssertionError(msg) def dbgMsg(myString): log.msg(myString) return 1 # these split_file_* functions are available for use as values to the # split_file= argument. def split_file_alwaystrunk(path): return (None, path) def split_file_branches(path): pieces = path.split('/') if pieces[0] == 'trunk': return (None, '/'.join(pieces[1:])) elif pieces[0] == 'branches': return (pieces[1], '/'.join(pieces[2:])) else: return None class SvnSource(base.ChangeSource, util.ComparableMixin): """This source will poll a perforce repository for changes and submit them to the change master.""" compare_attrs = ["svnuser", "svnpasswd", "svnurl", "pollinterval", "histmax"] parent = None # filled in when we're added last_change = None loop = None working = False def __init__(self, svnurl, split_file=None, svnuser=None, svnpasswd=None, svnbin='svn', pollinterval=10*60, histmax=100): """ @type svnurl: string @param svnurl: the SVN URL that describes the repository and subdirectory to watch. If this ChangeSource should only pay attention to a single branch, this should point at the repository for that branch, like svn://svn.twistedmatrix.com/svn/Twisted/trunk . If it should follow multiple branches, point it at the repository directory that contains all the branches like svn://svn.twistedmatrix.com/svn/Twisted and also provide a branch-determining function. Each file in the repository has a SVN URL in the form (SVNURL)/(BRANCH)/(FILEPATH), where (BRANCH) could be empty or not, depending upon your branch-determining function. Only files that start with (SVNURL)/(BRANCH) will be monitored. The Change objects that are sent to the Schedulers will see (FILEPATH) for each modified file. @type split_file: callable or None @param split_file: a function that is called with a string of the form (BRANCH)/(FILEPATH) and should return a tuple (BRANCH, FILEPATH). This function should match your repository's branch-naming policy. Each changed file has a fully-qualified URL that can be split into a prefix (which equals the value of the 'svnurl' argument) and a suffix; it is this suffix which is passed to the split_file function. If the function returns None, the file is ignored. Use this to indicate that the file is not a part of this project. For example, if your repository puts the trunk in trunk/... and branches are in places like branches/1.5/..., your split_file function could look like the following (this function is available as svnpoller.split_file_branches):: pieces = path.split('/') if pieces[0] == 'trunk': return (None, '/'.join(pieces[1:])) elif pieces[0] == 'branches': return (pieces[1], '/'.join(pieces[2:])) else: return None If instead your repository layout puts the trunk for ProjectA in trunk/ProjectA/... and the 1.5 branch in branches/1.5/ProjectA/..., your split_file function could look like:: pieces = path.split('/') if pieces[0] == 'trunk': branch = None pieces.pop(0) # remove 'trunk' elif pieces[0] == 'branches': pieces.pop(0) # remove 'branches' branch = pieces.pop(0) # grab branch name else: return None # something weird projectname = pieces.pop(0) if projectname != 'ProjectA': return None # wrong project return (branch, '/'.join(pieces)) The default of split_file= is None, which indicates that no splitting should be done. This is equivalent to the following function:: return (None, path) If you wish, you can override the split_file method with the same sort of function instead of passing in a split_file= argument. @type svnuser: string @param svnuser: If set, the --username option will be added to the 'svn log' command. You may need this to get access to a private repository. @type svnpasswd: string @param svnpasswd: If set, the --password option will be added. @type svnbin: string @param svnbin: path to svn binary, defaults to just 'svn'. Use this if your subversion command lives in an unusual location. @type pollinterval: int @param pollinterval: interval in seconds between polls. The default is 600 seconds (10 minutes). Smaller values decrease the latency between the time a change is recorded and the time the buildbot notices it, but it also increases the system load. @type histmax: int @param histmax: maximum number of changes to look back through. The default is 100. Smaller values decrease system load, but if more than histmax changes are recorded between polls, the extra ones will be silently lost. """ if svnurl.endswith("/"): svnurl = svnurl[:-1] # strip the trailing slash self.svnurl = svnurl self.split_file_function = split_file or split_file_alwaystrunk self.svnuser = svnuser self.svnpasswd = svnpasswd self.svnbin = svnbin self.pollinterval = pollinterval self.histmax = histmax self._root = None self.loop = LoopingCall(self.checksvn) def split_file(self, path): # use getattr() to avoid turning this function into a bound method, # which would require it to have an extra 'self' argument f = getattr(self, "split_file_function") return f(path) def startService(self): base.ChangeSource.startService(self) # Don't start the loop just yet because the reactor isn't running. # Give it a chance to go and install our SIGCHLD handler before # spawning processes. reactor.callLater(0, self.loop.start, self.pollinterval) def stopService(self): self.loop.stop() return base.ChangeSource.stopService(self) def describe(self): return "svnsource %s branch %s" % ( self.svnurl, self.branch) def checksvn(self): # Our return value is only used for unit testing. # we need to figure out the repository root, so we can figure out # repository-relative pathnames later. Each SVNURL is in the form # (ROOT)/(PROJECT)/(BRANCH)/(FILEPATH), where (ROOT) is something # like svn://svn.twistedmatrix.com/svn/Twisted (i.e. there is a # physical repository at /svn/Twisted on that host), (PROJECT) is # something like Projects/Twisted (i.e. within the repository's # internal namespace, everything under Projects/Twisted/ has # something to do with Twisted, but these directory names do not # actually appear on the repository host), (BRANCH) is something like # "trunk" or "branches/2.0.x", and (FILEPATH) is a tree-relative # filename like "twisted/internet/defer.py". # our self.svnurl attribute contains (ROOT)/(PROJECT) combined # together in a way that we can't separate without svn's help. If the # user is not using the split_file= argument, then self.svnurl might # be (ROOT)/(PROJECT)/(BRANCH) . In any case, the filenames we will # get back from 'svn log' will be of the form # (PROJECT)/(BRANCH)/(FILEPATH), but we want to be able to remove # that (PROJECT) prefix from them. To do this without requiring the # user to tell us how svnurl is split into ROOT and PROJECT, we do an # 'svn info --xml' command at startup. This command will include a # element that tells us ROOT. We then strip this prefix from # self.svnurl to determine PROJECT, and then later we strip the # PROJECT prefix from the filenames reported by 'svn log --xml' to # get a (BRANCH)/(FILEPATH) that can be passed to split_file() to # turn into separate BRANCH and FILEPATH values. # whew. if not self._prefix: # this sets self._prefix when it finishes d = self._determine_prefix() else: d = defer.succeed(None) if self.working: dbgMsg("Skipping checksvn because last one has not finished") return d self.working = True d.addCallback(lambda res: self._get_logs()) d.addCallback(self._parse_logs) d.addCallback(self._get_new_logentries) d.addCallback(self._create_changes) d.addCallback(self._submit_changes) d.addBoth(self._finished) return d def _determine_prefix(self): args = ["info", "--xml", "--non-interactive", self.svnurl] if self.svnuser: args.extend(["--username=%s" % self.svnuser]) if self.svnpasswd: args.extend(["--password=%s" % self.svnpasswd]) d = getProcessOutput(self.svnbin, args, {}) d.addCallback(self._determine_prefix_2) return d def _determine_prefix_2(self, output): try: doc = xml.dom.minidom.parseString(output) except xml.parsers.expat.ExpatError: dbgMsg("_process_changes: ExpatError in %s" % output) log.msg("SvnSource._determine_prefix_2: ExpatError in '%s'" % output) raise rootnodes = doc.getElementsByTagName("root") if not rootnodes: # this happens if the URL we gave was already the root. In this # case, our prefix is empty. self._prefix = "" return self._prefix rootnode = rootnodes[0] root = "".join([c.data for c in rootnode.childNodes]) # root will be a unicode string _assert(self.svnurl.startswith(root), "svnurl='%s' doesn't start with ='%s'" % (self.svnurl, root)) self._prefix = self.svnurl[len(root):] if self._prefix.startswith("/"): self._prefix = self._prefix[1:] log.msg("SvnSource: svnurl=%s, root=%s, so prefix=%s" % (self.svnurl, root, self._prefix)) return self._prefix def _get_logs(self): args = [] args.extend(["log", "--xml", "--verbose", "--non-interactive"]) if self.svnuser: args.extend(["--username=%s" % self.svnuser]) if self.svnpasswd: args.extend(["--password=%s" % self.svnpasswd]) args.extend(["--limit=%d" % (self.histmax), self.svnurl]) env = {} res = getProcessOutput(self.svnbin, args, env) log.msg("_get_changes args %s end %s returns %s" % (args, env, res)) return res def _parse_logs(self, output): # parse the XML output, return a list of nodes try: doc = xml.dom.minidom.parseString(output) except xml.parsers.expat.ExpatError: dbgMsg("_process_changes: ExpatError in %s" % output) log.msg("SvnSource._parse_changes: ExpatError in '%s'" % output) raise logentries = doc.getElementsByTagName("logentry") return logentries def _filter_new_logentries(self, logentries, last_change): # given a list of logentries, return a tuple of (new_last_change, # new_logentries), where new_logentries contains only the ones after # last_change if not logentries: # no entries, so last_change must stay at None return (None, []) mostRecent = int(logentries[0].getAttribute("revision")) if last_change is None: # if this is the first time we've been run, ignore any changes # that occurred before now. This prevents a build at every # startup. log.msg('svnPoller: starting at change %s' % mostRecent) self.last_change = mostRecent return (mostRecent, []) if last_change == mostRecent: # an unmodified repository will hit this case log.msg('svnPoller: _process_changes last %s mostRecent %s' % ( last_change, mostRecent)) return (mostRecent, []) new_logentries = [] for el in logentries: if last_change == int(el.getAttribute("revision")): break new_logentries.append(el) new_logentries.reverse() # return oldest first return (mostRecent, new_logentries) def _get_text(self, element, tag_name): child_nodes = element.getElementsByTagName(tag_name)[0].childNodes text = "".join([t.data for t in child_nodes]) return text def _transform_path(self, path): _assert(path.startswith(self._prefix), "filepath '%s' should start with prefix '%s'" % (path, self._prefix)) relative_path = path[len(self._prefix):] if relative_path.startswith("/"): relative_path = relative_path[1:] branch, final_path = self.split_file(relative_path) return branch, final_path def _get_new_logentries(self, logentries): last_change = self.last_change (new_last_change, new_logentries) = self._filter_new_logentries(logentries, self.last_change) self.last_change = new_last_change log.msg('svnPoller: _process_changes %s .. %s' % (last_change, new_last_change)) return new_logentries def _create_changes(self, new_logentries): changes = [] for el in new_logentries: branch_files = [] # get oldest change first revision = int(el.getAttribute("revision")) dbgMsg("Adding change revision %s" % (revision,)) author = self._get_text(el, "author") comments = self._get_text(el, "msg") when = self._get_text(el, "date") when = time.mktime(time.strptime("%.19s" % when, "%Y-%m-%dT%H:%M:%S")) branches = {} pathlist = el.getElementsByTagName("paths")[0] for p in pathlist.getElementsByTagName("path"): path = "".join([t.data for t in p.childNodes]) if path.startswith("/"): path = path[1:] branch, filename = self._transform_path(path) if not branch in branches: branches[branch] = [] branches[branch].append(filename) for branch in branches: c = Change(who=author, files=branches[branch], comments=comments, revision=revision, when=when, branch=branch) changes.append(c) return changes def _submit_changes(self, changes): for c in changes: self.parent.addChange(c) def _finished(self, res): dbgMsg('_finished : %s' % res) assert self.working self.working = False # Again, the return value is only for unit testing. # If there's a failure, log it so it isn't lost. if isinstance(res, failure.Failure): dbgMsg('checksvn failed: %s' % res) return res From warner at users.sourceforge.net Mon Oct 2 00:13:14 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:14 +0000 Subject: [Buildbot-commits] buildbot/buildbot/test test_svnpoller.py, NONE, 1.1 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/test In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17108/buildbot/test Added Files: test_svnpoller.py Log Message: [project @ svnpoller: add new ChangeSource, interim checkin] Original author: warner at lothar.com Date: 2006-10-01 17:18:18 --- NEW FILE: test_svnpoller.py --- # Here are the tests for the SvnSource import sys import time from twisted.python import log, failure from twisted.internet import defer from twisted.trial import unittest from buildbot.twcompat import maybeWait from buildbot.changes.changes import Change from buildbot.changes.svnpoller import SvnSource, split_file_branches # was changes 1012 in xenomai.org svn_change_1 = """ rpm 2006-05-17T14:58:28.494960Z /branch/ksrc/arch/i386/hal.c Remove unused variable """ svn_change_2 = """ rpm 2006-05-15T12:54:08.891420Z /trunk/ChangeLog /trunk/ksrc/first_file /trunk/ksrc/second_file Initial Adeos support """ svn_change_3 = """ rpm 2006-05-15T12:54:08.891420Z /trunk/ChangeLog /trunk/ksrc/first_file /trunk/ksrc/second_file Upgrade Adeos support """ def dbgMsg(myString): log.msg(myString) return 1 class MockSvnSource(SvnSource): """Test SvnSource which doesn't actually invoke svn.""" invocation = 0 def __init__(self, svnchanges, *args, **kwargs): SvnSource.__init__(self, None, *args, **kwargs) self.svnchanges = svnchanges def _get_changes(self): assert self.working result = self.svnchanges[self.invocation] self.invocation += 1 #log.msg("MockSvnSource._get_changes %s result %s " % (self.invocation-1, result)) dbgMsg("MockSvnSource._get_changes %s " % (self.invocation-1)) return defer.succeed(result) def _get_describe(self, dummy, num): assert self.working dbgMsg("MockSvnSource._get_describe %s " % num) return defer.succeed(self.svnchanges[num]) class TestSvnPoller(unittest.TestCase): def setUp(self): self.changes = [] self.addChange = self.changes.append def failUnlessIn(self, substr, string): # this is for compatibility with python2.2 if isinstance(string, str): self.failUnless(string.find(substr) != -1) else: self.assertIn(substr, string) def testCheck(self): """successful checks""" self.t = MockSvnSource(svnchanges=[ svn_change_1, svn_change_2, svn_change_3], svnuser=None, svnurl='/trunk/',) self.t.parent = self # The first time, it just learns the change to start at self.assert_(self.t.last_change is None) self.assert_(not self.t.working) dbgMsg("zzz") return maybeWait(self.t.checksvn().addCallback(self._testCheck2)) def _testCheck2(self, res): dbgMsg("zzz1") self.assertEquals(self.changes, []) dbgMsg("zzz2 %s %s" % (self.t.last_change, self.changes)) self.assertEquals(self.t.last_change, '101') dbgMsg("zzz3") # Subsequent times, it returns Change objects for new changes. return self.t.checksvn().addCallback(self._testCheck3) def _testCheck3(self, res): # They're supposed to go oldest to newest, so this one must be first. tstChange1 = Change(who='rpm', files=['/trunk/ChangeLog', '/trunk/ksrc/first_file', '/trunk/ksrc/second_file'], comments="Initial Adeos support", revision='2', when=self.makeTime("2006/05/15 12:54:08"), branch='trunk').asText() dbgMsg("tstChange" + tstChange1) dbgMsg("changes[0]" + self.changes[0].asText()) self.assertEquals(self.changes[0].asText(), tstChange1) # Subsequent times, it returns Change objects for new changes. return self.t.checksvn().addCallback(self._testCheck4) def _testCheck4(self, res): dbgMsg("zzz5 %s " % len(self.changes)) self.assertEquals(len(self.changes), 1) dbgMsg("zzz6 %s %s" % (self.t.last_change, self.changes)) self.assertEquals(self.t.last_change, '102') dbgMsg("zzz7") self.assert_(not self.t.working) tstChange2 = Change(who='rpm', files=['/trunk/ChangeLog', '/trunk/ksrc/first_file', '/trunk/ksrc/second_file'], comments="Initial Adeos support", revision='2', when=self.makeTime("2006/05/15 12:54:08"), branch='trunk').asText() dbgMsg("changes[0]" + self.changes[0].asText()) dbgMsg("tstChange2" + tstChange2) self.assertEquals(self.changes[0].asText(), tstChange2) dbgMsg(7777) def makeTime(self, timestring): datefmt = '%Y/%m/%d %H:%M:%S' when = time.mktime(time.strptime(timestring, datefmt)) return when def testFailedChanges(self): """'svn changes' failure is properly reported""" self.t = MockSvnSource(svnchanges=['Subversion client error:\n...'], svnuser=None, svnurl="/trunk") self.t.parent = self d = self.t.checksvn() d.addBoth(self._testFailedChanges2) return maybeWait(d) def _testFailedChanges2(self, f): self.assert_(isinstance(f, failure.Failure)) self.failUnlessIn('Subversion client error', str(f)) self.assert_(not self.t.working) def testFailedDescribe(self): """'svn describe' failure is properly reported""" self.t = MockSvnSource(svnchanges=[ svn_change_1, 'Subversion client error:\n...', svn_change_2,], svnuser=None) self.t.parent = self d = self.t.checksvn() dbgMsg("xxx") d.addCallback(self._testFailedDescribe2) return maybeWait(d) def _testFailedDescribe2(self, res): # first time finds nothing; check again. dbgMsg("yy") res = self.t.checksvn().addBoth(self._testFailedDescribe3) return res def _testFailedDescribe3(self, f): dbgMsg("yy1 %s" % f) self.assert_(isinstance(f, failure.Failure)) dbgMsg("yy2") self.failUnlessIn('Subversion client error', str(f)) dbgMsg("yy3") self.assert_(not self.t.working) dbgMsg("yy4") self.assertEquals(self.t.last_change, '101') dbgMsg("yy5") def testAlreadyWorking(self): """don't launch a new poll while old is still going""" self.t = SvnSource() self.t.working = True self.assert_(self.t.last_change is None) d = self.t.checksvn() d.addCallback(self._testAlreadyWorking2) def _testAlreadyWorking2(self, res): self.assert_(self.t.last_change is None) # this is the output of "svn info --xml # svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk" prefix_output = """\ svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk svn+ssh://svn.twistedmatrix.com/svn/Twisted bbbe8e31-12d6-0310-92fd-ac37d47ddeeb jml 2006-10-01T02:37:34.063255Z """ # and this is "svn info --xml svn://svn.twistedmatrix.com/svn/Twisted". I # think this is kind of a degenerate case.. it might even be a form of error. prefix_output_2 = """\ """ # this is the svn info output for a local repository, svn info --xml # file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository prefix_output_3 = """\ file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository c0f47ff4-ba1e-0410-96b5-d44cc5c79e7f warner 2006-10-01T07:37:04.182499Z """ # % svn info --xml file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample/trunk prefix_output_4 = """\ file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample/trunk file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository c0f47ff4-ba1e-0410-96b5-d44cc5c79e7f warner 2006-10-01T07:37:02.286440Z """ class ComputePrefix(unittest.TestCase): def test1(self): base = "svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk" s = SvnSource(base + "/") self.failUnlessEqual(s.svnurl, base) # certify slash-stripping prefix = s._determine_prefix_2(prefix_output) self.failUnlessEqual(prefix, "trunk") self.failUnlessEqual(s._prefix, prefix) def test2(self): base = "svn+ssh://svn.twistedmatrix.com/svn/Twisted" s = SvnSource(base) self.failUnlessEqual(s.svnurl, base) prefix = s._determine_prefix_2(prefix_output_2) self.failUnlessEqual(prefix, "") def test3(self): base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository" s = SvnSource(base) self.failUnlessEqual(s.svnurl, base) prefix = s._determine_prefix_2(prefix_output_3) self.failUnlessEqual(prefix, "") def test4(self): base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample/trunk" s = SvnSource(base) self.failUnlessEqual(s.svnurl, base) prefix = s._determine_prefix_2(prefix_output_4) self.failUnlessEqual(prefix, "sample/trunk") # output from svn log on .../SVN-Repository/sample # (so it includes trunk and branches) changes_output_1 = """\ warner 2006-10-01T07:37:04.182499Z /sample/branch/main.c commit_on_branch warner 2006-10-01T07:37:03.175326Z /sample/branch make_branch """ class ComputeChanges(unittest.TestCase): def test1(self): base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample" s = SvnSource(base) s._prefix = "sample" doc = s._parse_logs(changes_output_1) newlast, logentries = s._filter_new_logentries(doc, 3) self.failUnlessEqual(newlast, 3) self.failUnlessEqual(len(logentries), 0) newlast, logentries = s._filter_new_logentries(doc, 2) self.failUnlessEqual(newlast, 3) self.failUnlessEqual(len(logentries), 1) newlast, logentries = s._filter_new_logentries(doc, 0) self.failUnlessEqual(newlast, 3) self.failUnlessEqual(len(logentries), 2) def split_file(self, path): pieces = path.split("/") if pieces[0] == "branch": return "branch", "/".join(pieces[1:]) if pieces[0] == "trunk": return None, "/".join(pieces[1:]) raise RuntimeError("there shouldn't be any files like %s" % path) def testChanges(self): base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample" s = SvnSource(base, split_file=self.split_file) s._prefix = "sample" doc = s._parse_logs(changes_output_1) newlast, logentries = s._filter_new_logentries(doc, 0) changes = s._create_changes(logentries) self.failUnlessEqual(len(changes), 2) self.failUnlessEqual(changes[0].branch, "branch") self.failUnlessEqual(changes[1].branch, "branch") self.failUnlessEqual(changes[1].files, ["main.c"]) From warner at users.sourceforge.net Mon Oct 2 00:13:14 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:14 +0000 Subject: [Buildbot-commits] buildbot/docs buildbot.texinfo,1.81,1.82 Message-ID: Update of /cvsroot/buildbot/buildbot/docs In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17108/docs Modified Files: buildbot.texinfo Log Message: [project @ svnpoller: add new ChangeSource, interim checkin] Original author: warner at lothar.com Date: 2006-10-01 17:18:18 Index: buildbot.texinfo =================================================================== RCS file: /cvsroot/buildbot/buildbot/docs/buildbot.texinfo,v retrieving revision 1.81 retrieving revision 1.82 diff -u -d -r1.81 -r1.82 --- buildbot.texinfo 30 Sep 2006 21:12:08 -0000 1.81 +++ buildbot.texinfo 2 Oct 2006 00:13:12 -0000 1.82 @@ -153,6 +153,7 @@ * PBChangeSource:: * P4Source:: * BonsaiPoller:: +* SvnSource:: Build Process @@ -2525,6 +2526,7 @@ a suitable @code{MaildirSource}. +* SvnSource:: @menu * Choosing ChangeSources:: * CVSToys - PBService:: @@ -2533,6 +2535,7 @@ * PBChangeSource:: * P4Source:: * BonsaiPoller:: +* SvnSource:: @end menu @node Choosing ChangeSources, CVSToys - PBService, Change Sources, Change Sources @@ -2754,7 +2757,6 @@ @end table - @node P4Source, BonsaiPoller, PBChangeSource, Change Sources @subsection P4Source @@ -2806,7 +2808,7 @@ )) @end example - at node BonsaiPoller, , P4Source, Change Sources + at node BonsaiPoller, SvnSource, P4Source, Change Sources @subsection BonsaiPoller @csindex buildbot.changes.bonsaipoller.BonsaiPoller @@ -2823,6 +2825,59 @@ arguments it accepts. + at node SvnSource, , BonsaiPoller, Change Sources + at subsection SvnSource + + at csindex buildbot.changes.svnpoller.SvnSource + +The @code{SvnSource} periodically polls a + at uref{http://subversion.tigris.org/, Subversion} depot for changes, by +running the @code{svn} command in a subshell. It accepts the following +arguments: + + at table @samp + at item @code{svnurl} +The base URL path to watch. The repository this references could be +local or remote, depending upon the URL's prefix. + + at item @code{branch} +The branch to watch. Defaults to '/trunk'. + + at item @code{svnuser} +The Subversion user. + + at item @code{svnpasswd} +The Subversion user's password. + + at item @code{pollinterval} +How often to poll, in seconds. Defaults to 600 (10 minutes). + + at item @code{histmax} +The maximum number of changes to inspect at a time. If more than this +number occur since the last poll, older changes will be silently +ignored. Larger values of histmax will cause more time and memory to +be consumed on each poll attempt. @code{histmax} defaults to 100. + at end table + + at heading Example + +This example watches the Subversion SVN tree itself for changes. It +only pays attention to changes that appear on the trunk, and it labels +all of its Change objects as being on branch=None (which is the +convention within the buildbot for trunk changes). + + at example +from buildbot.changes.svnpoller import SvnSource +ss = SvnSource('http://svn.collab.net/repos/svn/trunk') +c['sources'].append(ss) + at end example + +This configuration uses the @code{SvnPORT}, @code{SvnUSER}, and + at code{SvnPASSWD} specified in the buildmaster's environment. It +watches a project in which the branch name is simply the next path +component, and the file is all path components after. + + @node Build Process, Status Delivery, Getting Source Code Changes, Top @chapter Build Process From warner at users.sourceforge.net Mon Oct 2 00:13:20 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:20 +0000 Subject: [Buildbot-commits] buildbot/buildbot/changes svnpoller.py,1.1,1.2 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/changes In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17137/buildbot/changes Modified Files: svnpoller.py Log Message: [project @ svnpoller: lots of work, make branches behave properly, write lots of tests, docs] Original author: warner at lothar.com Date: 2006-10-01 23:23:25 Index: svnpoller.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/changes/svnpoller.py,v retrieving revision 1.1 retrieving revision 1.2 diff -u -d -r1.1 -r1.2 --- svnpoller.py 2 Oct 2006 00:13:11 -0000 1.1 +++ svnpoller.py 2 Oct 2006 00:13:18 -0000 1.2 @@ -2,12 +2,12 @@ # Based on the work of Dave Peticolas for the P4poll # Changed to svn (using xml.dom.minidom) by Niklaus Giger +# Hacked beyond recognition by Brian Warner import time -from twisted.python import log, failure -from twisted.internet import defer, reactor -from twisted.internet.utils import getProcessOutput +from twisted.python import log +from twisted.internet import defer, reactor, utils from twisted.internet.task import LoopingCall from buildbot import util @@ -31,21 +31,25 @@ return (None, path) def split_file_branches(path): + # turn trunk/subdir/file.c into (None, "subdir/file.c") + # and branches/1.5.x/subdir/file.c into ("branches/1.5.x", "subdir/file.c") pieces = path.split('/') if pieces[0] == 'trunk': return (None, '/'.join(pieces[1:])) elif pieces[0] == 'branches': - return (pieces[1], '/'.join(pieces[2:])) + return ('/'.join(pieces[0:2]), '/'.join(pieces[2:])) else: return None class SvnSource(base.ChangeSource, util.ComparableMixin): - """This source will poll a perforce repository for changes and submit + """This source will poll a Subversion repository for changes and submit them to the change master.""" - compare_attrs = ["svnuser", "svnpasswd", "svnurl", - "pollinterval", "histmax"] + compare_attrs = ["svnurl", "split_file_function", + "svnuser", "svnpasswd", + "pollinterval", "histmax", + "svnbin"] parent = None # filled in when we're added last_change = None @@ -54,8 +58,8 @@ def __init__(self, svnurl, split_file=None, svnuser=None, svnpasswd=None, - svnbin='svn', - pollinterval=10*60, histmax=100): + pollinterval=10*60, histmax=100, + svnbin='svn'): """ @type svnurl: string @param svnurl: the SVN URL that describes the repository and @@ -100,7 +104,8 @@ if pieces[0] == 'trunk': return (None, '/'.join(pieces[1:])) elif pieces[0] == 'branches': - return (pieces[1], '/'.join(pieces[2:])) + return ('/'.join(pieces[0:2]), + '/'.join(pieces[2:])) else: return None @@ -115,7 +120,8 @@ pieces.pop(0) # remove 'trunk' elif pieces[0] == 'branches': pieces.pop(0) # remove 'branches' - branch = pieces.pop(0) # grab branch name + # grab branch name + branch = 'branches/' + pieces.pop(0) else: return None # something weird projectname = pieces.pop(0) @@ -141,11 +147,6 @@ @type svnpasswd: string @param svnpasswd: If set, the --password option will be added. - @type svnbin: string - @param svnbin: path to svn binary, defaults to just 'svn'. Use - this if your subversion command lives in an - unusual location. - @type pollinterval: int @param pollinterval: interval in seconds between polls. The default is 600 seconds (10 minutes). Smaller values @@ -159,6 +160,11 @@ system load, but if more than histmax changes are recorded between polls, the extra ones will be silently lost. + + @type svnbin: string + @param svnbin: path to svn binary, defaults to just 'svn'. Use + this if your subversion command lives in an + unusual location. """ if svnurl.endswith("/"): @@ -171,7 +177,8 @@ self.svnbin = svnbin self.pollinterval = pollinterval self.histmax = histmax - self._root = None + self._prefix = None + self.overrun_counter = 0 self.loop = LoopingCall(self.checksvn) def split_file(self, path): @@ -181,6 +188,7 @@ return f(path) def startService(self): + log.msg("SvnSource(%s) starting" % self.svnurl) base.ChangeSource.startService(self) # Don't start the loop just yet because the reactor isn't running. # Give it a chance to go and install our SIGCHLD handler before @@ -188,11 +196,12 @@ reactor.callLater(0, self.loop.start, self.pollinterval) def stopService(self): + log.msg("SvnSource(%s) shutting down" % self.svnurl) self.loop.stop() return base.ChangeSource.stopService(self) def describe(self): - return "svnsource %s branch %s" % ( self.svnurl, self.branch) + return "SvnSource watching %s" % self.svnurl def checksvn(self): # Our return value is only used for unit testing. @@ -226,43 +235,53 @@ # whew. + if self.working: + log.msg("SvnSource(%s) overrun: timer fired but the previous " + "poll had not yet finished.") + self.overrun_counter += 1 + return defer.succeed(None) + self.working = True + + log.msg("SvnSource polling") if not self._prefix: - # this sets self._prefix when it finishes - d = self._determine_prefix() + # this sets self._prefix when it finishes. It fires with + # self._prefix as well, because that makes the unit tests easier + # to write. + d = self.get_root() + d.addCallback(self.determine_prefix) else: - d = defer.succeed(None) + d = defer.succeed(self._prefix) - if self.working: - dbgMsg("Skipping checksvn because last one has not finished") - return d + d.addCallback(self.get_logs) + d.addCallback(self.parse_logs) + d.addCallback(self.get_new_logentries) + d.addCallback(self.create_changes) + d.addCallback(self.submit_changes) + d.addBoth(self.finished) + return d - self.working = True - d.addCallback(lambda res: self._get_logs()) - d.addCallback(self._parse_logs) - d.addCallback(self._get_new_logentries) - d.addCallback(self._create_changes) - d.addCallback(self._submit_changes) - d.addBoth(self._finished) + def getProcessOutput(self, args): + # this exists so we can override it during the unit tests + d = utils.getProcessOutput(self.svnbin, args, {}) return d - def _determine_prefix(self): + def get_root(self): args = ["info", "--xml", "--non-interactive", self.svnurl] if self.svnuser: args.extend(["--username=%s" % self.svnuser]) if self.svnpasswd: args.extend(["--password=%s" % self.svnpasswd]) - d = getProcessOutput(self.svnbin, args, {}) - d.addCallback(self._determine_prefix_2) + d = self.getProcessOutput(args) return d - def _determine_prefix_2(self, output): - try: - doc = xml.dom.minidom.parseString(output) - except xml.parsers.expat.ExpatError: - dbgMsg("_process_changes: ExpatError in %s" % output) + def determine_prefix(self, output): + try: + doc = xml.dom.minidom.parseString(output) + except xml.parsers.expat.ExpatError: + dbgMsg("_process_changes: ExpatError in %s" % output) log.msg("SvnSource._determine_prefix_2: ExpatError in '%s'" % output) - raise + raise rootnodes = doc.getElementsByTagName("root") if not rootnodes: # this happens if the URL we gave was already the root. In this @@ -282,7 +301,7 @@ (self.svnurl, root, self._prefix)) return self._prefix - def _get_logs(self): + def get_logs(self, ignored_prefix=None): args = [] args.extend(["log", "--xml", "--verbose", "--non-interactive"]) if self.svnuser: @@ -290,22 +309,21 @@ if self.svnpasswd: args.extend(["--password=%s" % self.svnpasswd]) args.extend(["--limit=%d" % (self.histmax), self.svnurl]) - env = {} - res = getProcessOutput(self.svnbin, args, env) - log.msg("_get_changes args %s end %s returns %s" % (args, env, res)) - return res + d = self.getProcessOutput(args) + return d - def _parse_logs(self, output): + def parse_logs(self, output): # parse the XML output, return a list of nodes - try: - doc = xml.dom.minidom.parseString(output) - except xml.parsers.expat.ExpatError: - dbgMsg("_process_changes: ExpatError in %s" % output) + try: + doc = xml.dom.minidom.parseString(output) + except xml.parsers.expat.ExpatError: + dbgMsg("_process_changes: ExpatError in %s" % output) log.msg("SvnSource._parse_changes: ExpatError in '%s'" % output) raise logentries = doc.getElementsByTagName("logentry") return logentries + def _filter_new_logentries(self, logentries, last_change): # given a list of logentries, return a tuple of (new_last_change, # new_logentries), where new_logentries contains only the ones after @@ -314,30 +332,40 @@ # no entries, so last_change must stay at None return (None, []) - mostRecent = int(logentries[0].getAttribute("revision")) + mostRecent = int(logentries[0].getAttribute("revision")) if last_change is None: # if this is the first time we've been run, ignore any changes # that occurred before now. This prevents a build at every # startup. - log.msg('svnPoller: starting at change %s' % mostRecent) - self.last_change = mostRecent - return (mostRecent, []) + log.msg('svnPoller: starting at change %s' % mostRecent) + return (mostRecent, []) - if last_change == mostRecent: + if last_change == mostRecent: # an unmodified repository will hit this case - log.msg('svnPoller: _process_changes last %s mostRecent %s' % ( - last_change, mostRecent)) + log.msg('svnPoller: _process_changes last %s mostRecent %s' % ( + last_change, mostRecent)) return (mostRecent, []) new_logentries = [] - for el in logentries: + for el in logentries: if last_change == int(el.getAttribute("revision")): - break + break new_logentries.append(el) new_logentries.reverse() # return oldest first return (mostRecent, new_logentries) + def get_new_logentries(self, logentries): + last_change = self.last_change + (new_last_change, + new_logentries) = self._filter_new_logentries(logentries, + self.last_change) + self.last_change = new_last_change + log.msg('svnPoller: _process_changes %s .. %s' % + (last_change, new_last_change)) + return new_logentries + + def _get_text(self, element, tag_name): child_nodes = element.getElementsByTagName(tag_name)[0].childNodes text = "".join([t.data for t in child_nodes]) @@ -350,41 +378,39 @@ relative_path = path[len(self._prefix):] if relative_path.startswith("/"): relative_path = relative_path[1:] - branch, final_path = self.split_file(relative_path) - return branch, final_path - - def _get_new_logentries(self, logentries): - last_change = self.last_change - (new_last_change, - new_logentries) = self._filter_new_logentries(logentries, - self.last_change) - self.last_change = new_last_change - log.msg('svnPoller: _process_changes %s .. %s' % - (last_change, new_last_change)) - return new_logentries + where = self.split_file(relative_path) + # 'where' is either None or (branch, final_path) + return where - def _create_changes(self, new_logentries): + def create_changes(self, new_logentries): changes = [] for el in new_logentries: - branch_files = [] # get oldest change first - revision = int(el.getAttribute("revision")) - dbgMsg("Adding change revision %s" % (revision,)) - author = self._get_text(el, "author") - comments = self._get_text(el, "msg") - when = self._get_text(el, "date") - when = time.mktime(time.strptime("%.19s" % when, + branch_files = [] # get oldest change first + # TODO: revisit this, I think I've settled on Change.revision + # being a string everywhere, and leaving the interpretation + # of that string up to b.s.source.SVN methods + revision = int(el.getAttribute("revision")) + dbgMsg("Adding change revision %s" % (revision,)) + author = self._get_text(el, "author") + comments = self._get_text(el, "msg") + when = self._get_text(el, "date") + when = time.mktime(time.strptime("%.19s" % when, "%Y-%m-%dT%H:%M:%S")) branches = {} - pathlist = el.getElementsByTagName("paths")[0] - for p in pathlist.getElementsByTagName("path"): - path = "".join([t.data for t in p.childNodes]) + pathlist = el.getElementsByTagName("paths")[0] + for p in pathlist.getElementsByTagName("path"): + path = "".join([t.data for t in p.childNodes]) if path.startswith("/"): path = path[1:] - branch, filename = self._transform_path(path) - if not branch in branches: - branches[branch] = [] - branches[branch].append(filename) + where = self._transform_path(path) + # if 'where' is None, the file was outside any project that + # we care about and we should ignore it + if where: + branch, filename = where + if not branch in branches: + branches[branch] = [] + branches[branch].append(filename) for branch in branches: c = Change(who=author, @@ -397,17 +423,13 @@ return changes - def _submit_changes(self, changes): + def submit_changes(self, changes): for c in changes: - self.parent.addChange(c) + self.parent.addChange(c) - def _finished(self, res): + def finished(self, res): + log.msg("SvnSource finished polling") dbgMsg('_finished : %s' % res) assert self.working self.working = False - - # Again, the return value is only for unit testing. - # If there's a failure, log it so it isn't lost. - if isinstance(res, failure.Failure): - dbgMsg('checksvn failed: %s' % res) return res From warner at users.sourceforge.net Mon Oct 2 00:13:20 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:20 +0000 Subject: [Buildbot-commits] buildbot/docs buildbot.texinfo,1.82,1.83 Message-ID: Update of /cvsroot/buildbot/buildbot/docs In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17137/docs Modified Files: buildbot.texinfo Log Message: [project @ svnpoller: lots of work, make branches behave properly, write lots of tests, docs] Original author: warner at lothar.com Date: 2006-10-01 23:23:25 Index: buildbot.texinfo =================================================================== RCS file: /cvsroot/buildbot/buildbot/docs/buildbot.texinfo,v retrieving revision 1.82 retrieving revision 1.83 diff -u -d -r1.82 -r1.83 --- buildbot.texinfo 2 Oct 2006 00:13:12 -0000 1.82 +++ buildbot.texinfo 2 Oct 2006 00:13:18 -0000 1.83 @@ -2830,52 +2830,274 @@ @csindex buildbot.changes.svnpoller.SvnSource -The @code{SvnSource} periodically polls a - at uref{http://subversion.tigris.org/, Subversion} depot for changes, by -running the @code{svn} command in a subshell. It accepts the following -arguments: +The @code{buildbot.changes.svnpoller.SvnSource} is a ChangeSource +which periodically polls a @uref{http://subversion.tigris.org/, +Subversion} repository for new revisions, by running the @code{svn +log} command in a subshell. It can watch a single branch or multiple +branches. - at table @samp - at item @code{svnurl} -The base URL path to watch. The repository this references could be -local or remote, depending upon the URL's prefix. + at code{SvnSource} accepts the following arguments: - at item @code{branch} -The branch to watch. Defaults to '/trunk'. + at table @code + at item svnurl +The base URL path to watch, like + at code{svn://svn.twistedmatrix.com/svn/Twisted/trunk}, or + at code{http://divmod.org/svn/Divmod/}, or even + at code{file:///home/svn/Repository/ProjectA/branches/1.5/}. This must +include the access scheme, the location of the repository (both the +hostname for remote ones, and any additional directory names necessary +to get to the repository), and the sub-path within the repository's +virtual filesystem for the project and branch of interest. - at item @code{svnuser} -The Subversion user. +The @code{SvnSource} will only pay attention to files inside the +subdirectory specified by the complete svnurl. - at item @code{svnpasswd} -The Subversion user's password. + at item split_file +A function to convert pathnames into (branch, relative_pathname) +tuples. Use this to explain your repository's branch-naming policy to + at code{SvnSource}. This function must accept a single string and return +a two-entry tuple. There are a few utility functions in + at code{buildbot.changes.svnpoller} that can be used as a + at code{split_file} function, see below for details. - at item @code{pollinterval} -How often to poll, in seconds. Defaults to 600 (10 minutes). +The default value always returns (None, path), which indicates that +all files are on the trunk. + +Subclasses of @code{SvnSource} can override the @code{split_file} +method instead of using the @code{split_file=} argument. + + at item svnuser +An optional string parameter. If set, the @code{--user} argument will +be added to all @code{svn} commands. Use this if you have to +authenticate to the svn server before you can do @code{svn info} or + at code{svn log} commands. + + at item svnpasswd +Like @code{svnuser}, this will cause a @code{--password} argument to +be passed to all svn commands. + + at item pollinterval +How often to poll, in seconds. Defaults to 600 (checking once every 10 +minutes). Lower this if you want the buildbot to notice changes +faster, raise it if you want to reduce the network and CPU load on +your svn server. Please be considerate of public SVN repositories by +using a large interval when polling them. + + at item histmax +The maximum number of changes to inspect at a time. Every POLLINTERVAL +seconds, the @code{SvnSource} asks for the last HISTMAX changes and +looks through them for any ones it does not already know about. If +more than HISTMAX revisions have been committed since the last poll, +older changes will be silently ignored. Larger values of histmax will +cause more time and memory to be consumed on each poll attempt. + at code{histmax} defaults to 100. + + at item svnbin +This controls the @code{svn} executable to use. If subversion is +installed in a weird place on your system (outside of the +buildmaster's @code{$PATH}), use this to tell @code{SvnSource} where +to find it. The default value of ``svn'' will almost always be +sufficient. - at item @code{histmax} -The maximum number of changes to inspect at a time. If more than this -number occur since the last poll, older changes will be silently -ignored. Larger values of histmax will cause more time and memory to -be consumed on each poll attempt. @code{histmax} defaults to 100. @end table - at heading Example + at heading Branches -This example watches the Subversion SVN tree itself for changes. It -only pays attention to changes that appear on the trunk, and it labels -all of its Change objects as being on branch=None (which is the -convention within the buildbot for trunk changes). +Each source file that is tracked by a Subversion repository has a +fully-qualified SVN URL in the following form: +(REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the + at code{SvnSource}, you give it a @code{svnurl} value that includes all +of the REPOURL and possibly some portion of the PROJECT-plus-BRANCH +string. The @code{SvnSource} is responsible for producing Changes that +contain a branch name and a FILEPATH (which is relative to the top of +a checked-out tree). The details of how these strings are split up +depend upon how your repository names its branches. + + at subheading PROJECT/BRANCHNAME/FILEPATH repositories + +One common layout is to have all the various projects that share a +repository get a single top-level directory each. Then under a given +project's directory, you get two subdirectories, one named ``trunk'' +and another named ``branches''. Under ``branches'' you have a bunch of +other directories, one per branch, with names like ``1.5.x'' and +``testing''. It is also common to see directories like ``tags'' and +``releases'' next to ``branches'' and ``trunk''. + +For example, the Twisted project has a subversion server on +``svn.twistedmatrix.com'' that hosts several sub-projects. The +repository is available through a SCHEME of ``svn:''. The primary +sub-project is Twisted, of course, with a repository root of +``svn://svn.twistedmatrix.com/svn/Twisted''. Another sub-project is +Informant, with a root of +``svn://svn.twistedmatrix.com/svn/Informant'', etc. Inside any +checked-out Twisted tree, there is a file named bin/trial (which is +used to run unit test suites). + +The trunk for Twisted is in +``svn://svn.twistedmatrix.com/svn/Twisted/trunk'', and the +fully-qualified SVN URL for the trunk version of @code{trial} would be +``svn://svn.twistedmatrix.com/svn/Twisted/trunk/bin/trial''. The same +SVNURL for that file on a branch named ``1.5.x'' would be +``svn://svn.twistedmatrix.com/svn/Twisted/branches/1.5.x/bin/trial''. + +To set up a @code{SvnSource} that watches the Twisted trunk (and +nothing else), we would use the following: @example from buildbot.changes.svnpoller import SvnSource -ss = SvnSource('http://svn.collab.net/repos/svn/trunk') +s = SvnSource("svn://svn.twistedmatrix.com/svn/Twisted/trunk") c['sources'].append(ss) @end example -This configuration uses the @code{SvnPORT}, @code{SvnUSER}, and - at code{SvnPASSWD} specified in the buildmaster's environment. It -watches a project in which the branch name is simply the next path -component, and the file is all path components after. +In this case, every Change that our @code{SvnSource} produces will +have @code{.branch=None}, to indicate that the Change is on the trunk. +No other sub-projects or branches will be tracked. + +If we want our ChangeSource to follow multiple branches, we have to do +two things. First we have to change our @code{svnurl=} argument to +watch more than just ``.../Twisted/trunk''. We will set it to +``.../Twisted'' so that we'll see both the trunk and all the branches. +Second, we have to tell @code{SvnSource} how to split the +(PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository +out into (BRANCH) and (FILEPATH) pairs. + +We do the latter by providing a ``split_file'' function. This function +is responsible for splitting something like +``branches/1.5.x/bin/trial'' into @code{branch}=''branches/1.5.x'' and + at code{filepath}=''bin/trial''. This function is always given a string +that names a file relative to the subdirectory pointed to by the + at code{SvnSource}'s @code{svnurl=} argument. It is expected to return a +(BRANCHNAME, FILEPATH) tuple (in which FILEPATH is relative to the +branch indicated), or None to indicate that the file is outside any +project of interest. + +(note that we want to see ``branches/1.5.x'' rather than just +``1.5.x'' because when we perform the SVN checkout, we will probably +append the branch name to the baseURL, which requires that we keep the +``branches'' component in there. Other VC schemes use a different +approach towards branches and may not require this artifact.) + +If your repository uses this same PROJECT/BRANCH/FILEPATH naming +scheme, the following function will work: + + at example +def split_file_branches(path): + pieces = path.split('/') + if pieces[0] == 'trunk': + return (None, '/'.join(pieces[1:])) + elif pieces[0] == 'branches': + return ('/'.join(pieces[0:2]), + '/'.join(pieces[2:])) + else: + return None + at end example + +This function is provided as + at code{buildbot.changes.svnpoller.split_file_branches} for your +convenience. So to have our Twisted-watching @code{SvnSource} follow +multiple branches, we would use this: + + at example +from buildbot.changes.svnpoller import SvnSource, split_file_branches +s = SvnSource("svn://svn.twistedmatrix.com/svn/Twisted", + split_file=split_file_branches) +c['sources'].append(ss) + at end example + +Changes for all sorts of branches (with names like ``branches/1.5.x'', +and None to indicate the trunk) will be delivered to the Schedulers. +Each Scheduler is then free to use or ignore each branch as it sees +fit. + + at subheading BRANCHNAME/PROJECT/FILEPATH repositories + +Another common way to organize a Subversion repository is to put the +branch name at the top, and the projects underneath. This is +especially frequent when there are a number of related sub-projects +that all get released in a group. + +For example, Divmod.org hosts a project named ``Nevow'' as well as one +named ``Quotient''. In a checked-out Nevow tree there is a directory +named ``formless'' that contains a python source file named +``webform.py''. This repository is accessible via webdav (and thus +uses an ``http:'' scheme) through the divmod.org hostname. There are +many branches in this repository, and they use a +(BRANCHNAME)/(PROJECT) naming policy. + +The fully-qualified SVN URL for the trunk version of webform.py is + at code{http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py}. +You can do an @code{svn co} with that URL and get a copy of the latest +version. The 1.5.x branch version of this file would have a URL of + at code{http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py}. +The whole Nevow trunk would be checked out with + at code{http://divmod.org/svn/Divmod/trunk/Nevow}, while the Quotient +trunk would be checked out using + at code{http://divmod.org/svn/Divmod/trunk/Quotient}. + +Now suppose we want to have an @code{SvnSource} that only cares about +the Nevow trunk. This case looks just like the PROJECT/BRANCH layout +described earlier: + + at example +from buildbot.changes.svnpoller import SvnSource +s = SvnSource("http://divmod.org/svn/Divmod/trunk/Nevow") +c['sources'].append(ss) + at end example + +But what happens when we want to track multiple Nevow branches? We +have to point our @code{svnurl=} high enough to see all those +branches, but we also don't want to include Quotient changes (since +we're only building Nevow). To accomplish this, we must rely upon the + at code{split_file} function to help us tell the difference between +files that belong to Nevow and those that belong to Quotient, as well +as figuring out which branch each one is on. + + at example +from buildbot.changes.svnpoller import SvnSource +s = SvnSource("http://divmod.org/svn/Divmod", + split_file=my_file_splitter) +c['sources'].append(ss) + at end example + +The @code{my_file_splitter} function will be called with +repository-relative pathnames like: + + at table @code + at item trunk/Nevow/formless/webform.py +This is a Nevow file, on the trunk. We want the Change that includes this +to see a filename of @code{formless/webform.py"}, and a branch of None + + at item branches/1.5.x/Nevow/formless/webform.py +This is a Nevow file, on a branch. We want to get +branch=''branches/1.5.x'' and filename=''formless/webform.py''. + + at item trunk/Quotient/setup.py +This is a Quotient file, so we want to ignore it by having + at code{my_file_splitter} return None. + + at item branches/1.5.x/Quotient/setup.py +This is also a Quotient file, which should be ignored. + at end table + +The following definition for @code{my_file_splitter} will do the job: + + at example +def my_file_splitter(path): + pieces = path.split('/') + if pieces[0] == 'trunk': + branch = None + pieces.pop(0) # remove 'trunk' + elif pieces[0] == 'branches': + pieces.pop(0) # remove 'branches' + # grab branch name + branch = 'branches/' + pieces.pop(0) + else: + return None # something weird + projectname = pieces.pop(0) + if projectname != 'Nevow': + return None # wrong project + return (branch, '/'.join(pieces)) + at end example @node Build Process, Status Delivery, Getting Source Code Changes, Top From warner at users.sourceforge.net Mon Oct 2 00:13:20 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:20 +0000 Subject: [Buildbot-commits] buildbot/buildbot/test test_svnpoller.py, 1.1, 1.2 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/test In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17137/buildbot/test Modified Files: test_svnpoller.py Log Message: [project @ svnpoller: lots of work, make branches behave properly, write lots of tests, docs] Original author: warner at lothar.com Date: 2006-10-01 23:23:25 Index: test_svnpoller.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_svnpoller.py,v retrieving revision 1.1 retrieving revision 1.2 diff -u -d -r1.1 -r1.2 --- test_svnpoller.py 2 Oct 2006 00:13:12 -0000 1.1 +++ test_svnpoller.py 2 Oct 2006 00:13:18 -0000 1.2 @@ -1,227 +1,9 @@ -# Here are the tests for the SvnSource +# -*- test-case-name: buildbot.test.test_svnpoller -*- -import sys import time - -from twisted.python import log, failure from twisted.internet import defer from twisted.trial import unittest - -from buildbot.twcompat import maybeWait -from buildbot.changes.changes import Change -from buildbot.changes.svnpoller import SvnSource, split_file_branches - -# was changes 1012 in xenomai.org -svn_change_1 = """ - - -rpm -2006-05-17T14:58:28.494960Z - -/branch/ksrc/arch/i386/hal.c - -Remove unused variable - - -""" - -svn_change_2 = """ - - -rpm -2006-05-15T12:54:08.891420Z - -/trunk/ChangeLog -/trunk/ksrc/first_file -/trunk/ksrc/second_file - -Initial Adeos support - - -""" - -svn_change_3 = """ - - -rpm -2006-05-15T12:54:08.891420Z - -/trunk/ChangeLog -/trunk/ksrc/first_file -/trunk/ksrc/second_file - -Upgrade Adeos support - - -""" - -def dbgMsg(myString): - log.msg(myString) - return 1 - -class MockSvnSource(SvnSource): - """Test SvnSource which doesn't actually invoke svn.""" - invocation = 0 - - def __init__(self, svnchanges, *args, **kwargs): - SvnSource.__init__(self, None, *args, **kwargs) - self.svnchanges = svnchanges - - def _get_changes(self): - assert self.working - result = self.svnchanges[self.invocation] - self.invocation += 1 - #log.msg("MockSvnSource._get_changes %s result %s " % (self.invocation-1, result)) - dbgMsg("MockSvnSource._get_changes %s " % (self.invocation-1)) - return defer.succeed(result) - - def _get_describe(self, dummy, num): - assert self.working - dbgMsg("MockSvnSource._get_describe %s " % num) - return defer.succeed(self.svnchanges[num]) - -class TestSvnPoller(unittest.TestCase): - def setUp(self): - self.changes = [] - self.addChange = self.changes.append - - def failUnlessIn(self, substr, string): - # this is for compatibility with python2.2 - if isinstance(string, str): - self.failUnless(string.find(substr) != -1) - else: - self.assertIn(substr, string) - - def testCheck(self): - """successful checks""" - self.t = MockSvnSource(svnchanges=[ svn_change_1, svn_change_2, svn_change_3], - svnuser=None, - svnurl='/trunk/',) - self.t.parent = self - - # The first time, it just learns the change to start at - self.assert_(self.t.last_change is None) - self.assert_(not self.t.working) - dbgMsg("zzz") - return maybeWait(self.t.checksvn().addCallback(self._testCheck2)) - - def _testCheck2(self, res): - dbgMsg("zzz1") - self.assertEquals(self.changes, []) - dbgMsg("zzz2 %s %s" % (self.t.last_change, self.changes)) - self.assertEquals(self.t.last_change, '101') - dbgMsg("zzz3") - - # Subsequent times, it returns Change objects for new changes. - return self.t.checksvn().addCallback(self._testCheck3) - - def _testCheck3(self, res): - # They're supposed to go oldest to newest, so this one must be first. - tstChange1 = Change(who='rpm', - files=['/trunk/ChangeLog', - '/trunk/ksrc/first_file', - '/trunk/ksrc/second_file'], - comments="Initial Adeos support", - revision='2', - when=self.makeTime("2006/05/15 12:54:08"), - branch='trunk').asText() - dbgMsg("tstChange" + tstChange1) - dbgMsg("changes[0]" + self.changes[0].asText()) - - self.assertEquals(self.changes[0].asText(), tstChange1) - # Subsequent times, it returns Change objects for new changes. - return self.t.checksvn().addCallback(self._testCheck4) - - def _testCheck4(self, res): - dbgMsg("zzz5 %s " % len(self.changes)) - self.assertEquals(len(self.changes), 1) - dbgMsg("zzz6 %s %s" % (self.t.last_change, self.changes)) - self.assertEquals(self.t.last_change, '102') - dbgMsg("zzz7") - self.assert_(not self.t.working) - tstChange2 = Change(who='rpm', - files=['/trunk/ChangeLog', - '/trunk/ksrc/first_file', - '/trunk/ksrc/second_file'], - comments="Initial Adeos support", - revision='2', - when=self.makeTime("2006/05/15 12:54:08"), - branch='trunk').asText() - dbgMsg("changes[0]" + self.changes[0].asText()) - dbgMsg("tstChange2" + tstChange2) - self.assertEquals(self.changes[0].asText(), tstChange2) - dbgMsg(7777) - - def makeTime(self, timestring): - datefmt = '%Y/%m/%d %H:%M:%S' - when = time.mktime(time.strptime(timestring, datefmt)) - return when - - def testFailedChanges(self): - """'svn changes' failure is properly reported""" - self.t = MockSvnSource(svnchanges=['Subversion client error:\n...'], - svnuser=None, - svnurl="/trunk") - self.t.parent = self - d = self.t.checksvn() - d.addBoth(self._testFailedChanges2) - return maybeWait(d) - - def _testFailedChanges2(self, f): - self.assert_(isinstance(f, failure.Failure)) - self.failUnlessIn('Subversion client error', str(f)) - self.assert_(not self.t.working) - - def testFailedDescribe(self): - """'svn describe' failure is properly reported""" - self.t = MockSvnSource(svnchanges=[ - svn_change_1, - 'Subversion client error:\n...', - svn_change_2,], - svnuser=None) - self.t.parent = self - d = self.t.checksvn() - dbgMsg("xxx") - d.addCallback(self._testFailedDescribe2) - return maybeWait(d) - - def _testFailedDescribe2(self, res): - # first time finds nothing; check again. - dbgMsg("yy") - res = self.t.checksvn().addBoth(self._testFailedDescribe3) - return res - - def _testFailedDescribe3(self, f): - dbgMsg("yy1 %s" % f) - self.assert_(isinstance(f, failure.Failure)) - dbgMsg("yy2") - self.failUnlessIn('Subversion client error', str(f)) - dbgMsg("yy3") - self.assert_(not self.t.working) - dbgMsg("yy4") - self.assertEquals(self.t.last_change, '101') - dbgMsg("yy5") - - def testAlreadyWorking(self): - """don't launch a new poll while old is still going""" - self.t = SvnSource() - self.t.working = True - self.assert_(self.t.last_change is None) - d = self.t.checksvn() - d.addCallback(self._testAlreadyWorking2) - - def _testAlreadyWorking2(self, res): - self.assert_(self.t.last_change is None) +from buildbot.changes.svnpoller import SvnSource # this is the output of "svn info --xml # svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk" @@ -307,7 +89,7 @@ base = "svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk" s = SvnSource(base + "/") self.failUnlessEqual(s.svnurl, base) # certify slash-stripping - prefix = s._determine_prefix_2(prefix_output) + prefix = s.determine_prefix(prefix_output) self.failUnlessEqual(prefix, "trunk") self.failUnlessEqual(s._prefix, prefix) @@ -315,42 +97,59 @@ base = "svn+ssh://svn.twistedmatrix.com/svn/Twisted" s = SvnSource(base) self.failUnlessEqual(s.svnurl, base) - prefix = s._determine_prefix_2(prefix_output_2) + prefix = s.determine_prefix(prefix_output_2) self.failUnlessEqual(prefix, "") def test3(self): base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository" s = SvnSource(base) self.failUnlessEqual(s.svnurl, base) - prefix = s._determine_prefix_2(prefix_output_3) + prefix = s.determine_prefix(prefix_output_3) self.failUnlessEqual(prefix, "") def test4(self): base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample/trunk" s = SvnSource(base) self.failUnlessEqual(s.svnurl, base) - prefix = s._determine_prefix_2(prefix_output_4) + prefix = s.determine_prefix(prefix_output_4) self.failUnlessEqual(prefix, "sample/trunk") # output from svn log on .../SVN-Repository/sample # (so it includes trunk and branches) -changes_output_1 = """\ - - +sample_base = "file:///usr/home/warner/stuff/Projects/BuildBot/trees/misc/_trial_temp/test_vc/repositories/SVN-Repository/sample" +sample_logentries = [None] * 4 + +sample_logentries[3] = """\ + +warner +2006-10-01T19:35:16.165664Z + +/sample/trunk/version.c + +revised_to_2 + +""" + +sample_logentries[2] = """\ warner -2006-10-01T07:37:04.182499Z +2006-10-01T19:35:10.215692Z /sample/branch/main.c commit_on_branch +""" + +sample_logentries[1] = """\ warner -2006-10-01T07:37:03.175326Z +2006-10-01T19:35:09.154973Z make_branch - """ +sample_logentries[0] = """\ + +warner +2006-10-01T19:35:08.642045Z + +/sample +/sample/trunk +/sample/trunk/subdir/subdir.c +/sample/trunk/main.c +/sample/trunk/version.c +/sample/trunk/subdir + +sample_project_files + +""" + +sample_info_output = """\ + + + +file:///usr/home/warner/stuff/Projects/BuildBot/trees/misc/_trial_temp/test_vc/repositories/SVN-Repository/sample + +file:///usr/home/warner/stuff/Projects/BuildBot/trees/misc/_trial_temp/test_vc/repositories/SVN-Repository +4f94adfc-c41e-0410-92d5-fbf86b7c7689 + + +warner +2006-10-01T19:35:16.165664Z + + + +""" + + +changes_output_template = """\ + + +%s +""" + +def make_changes_output(maxrevision): + # return what 'svn log' would have just after the given revision was + # committed + logs = sample_logentries[0:maxrevision] + assert len(logs) == maxrevision + logs.reverse() + output = changes_output_template % ("".join(logs)) + return output + +def split_file(path): + pieces = path.split("/") + if pieces[0] == "branch": + return "branch", "/".join(pieces[1:]) + if pieces[0] == "trunk": + return None, "/".join(pieces[1:]) + raise RuntimeError("there shouldn't be any files like %s" % path) + +class MySvnSource(SvnSource): + def __init__(self, *args, **kwargs): + SvnSource.__init__(self, *args, **kwargs) + self.pending_commands = [] + self.finished_changes = [] + + def getProcessOutput(self, args): + d = defer.Deferred() + self.pending_commands.append((args, d)) + return d + + def submit_changes(self, changes): + self.finished_changes.extend(changes) + class ComputeChanges(unittest.TestCase): def test1(self): base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample" s = SvnSource(base) s._prefix = "sample" - doc = s._parse_logs(changes_output_1) + output = make_changes_output(4) + doc = s.parse_logs(output) - newlast, logentries = s._filter_new_logentries(doc, 3) - self.failUnlessEqual(newlast, 3) + newlast, logentries = s._filter_new_logentries(doc, 4) + self.failUnlessEqual(newlast, 4) self.failUnlessEqual(len(logentries), 0) - newlast, logentries = s._filter_new_logentries(doc, 2) - self.failUnlessEqual(newlast, 3) + newlast, logentries = s._filter_new_logentries(doc, 3) + self.failUnlessEqual(newlast, 4) self.failUnlessEqual(len(logentries), 1) - newlast, logentries = s._filter_new_logentries(doc, 0) - self.failUnlessEqual(newlast, 3) - self.failUnlessEqual(len(logentries), 2) + newlast, logentries = s._filter_new_logentries(doc, 1) + self.failUnlessEqual(newlast, 4) + self.failUnlessEqual(len(logentries), 3) - def split_file(self, path): - pieces = path.split("/") - if pieces[0] == "branch": - return "branch", "/".join(pieces[1:]) - if pieces[0] == "trunk": - return None, "/".join(pieces[1:]) - raise RuntimeError("there shouldn't be any files like %s" % path) + newlast, logentries = s._filter_new_logentries(doc, None) + self.failUnlessEqual(newlast, 4) + self.failUnlessEqual(len(logentries), 0) def testChanges(self): base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample" - s = SvnSource(base, split_file=self.split_file) + s = SvnSource(base, split_file=split_file) s._prefix = "sample" - doc = s._parse_logs(changes_output_1) - newlast, logentries = s._filter_new_logentries(doc, 0) - changes = s._create_changes(logentries) + doc = s.parse_logs(make_changes_output(3)) + newlast, logentries = s._filter_new_logentries(doc, 1) + # so we see revisions 2 and 3 as being new + self.failUnlessEqual(newlast, 3) + changes = s.create_changes(logentries) self.failUnlessEqual(len(changes), 2) self.failUnlessEqual(changes[0].branch, "branch") + self.failUnlessEqual(changes[0].revision, 2) self.failUnlessEqual(changes[1].branch, "branch") self.failUnlessEqual(changes[1].files, ["main.c"]) + self.failUnlessEqual(changes[1].revision, 3) + + # and now pull in r4 + doc = s.parse_logs(make_changes_output(4)) + newlast, logentries = s._filter_new_logentries(doc, newlast) + self.failUnlessEqual(newlast, 4) + # so we see revision 4 as being new + changes = s.create_changes(logentries) + self.failUnlessEqual(len(changes), 1) + self.failUnlessEqual(changes[0].branch, None) + self.failUnlessEqual(changes[0].revision, 4) + self.failUnlessEqual(changes[0].files, ["version.c"]) + + def testFirstTime(self): + base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample" + s = SvnSource(base, split_file=split_file) + s._prefix = "sample" + doc = s.parse_logs(make_changes_output(4)) + logentries = s.get_new_logentries(doc) + # SvnSource ignores all changes that happened before it was started + self.failUnlessEqual(len(logentries), 0) + self.failUnlessEqual(s.last_change, 4) + +class Misc(unittest.TestCase): + def testAlreadyWorking(self): + base = "file:///home/warner/stuff/Projects/BuildBot/trees/svnpoller/_trial_temp/test_vc/repositories/SVN-Repository/sample" + s = MySvnSource(base) + d = s.checksvn() + # the SvnSource is now waiting for its getProcessOutput to finish + self.failUnlessEqual(s.overrun_counter, 0) + d2 = s.checksvn() + self.failUnlessEqual(s.overrun_counter, 1) + self.failUnlessEqual(len(s.pending_commands), 1) + + def testGetRoot(self): + base = "svn+ssh://svn.twistedmatrix.com/svn/Twisted/trunk" + s = MySvnSource(base) + d = s.checksvn() + # the SvnSource is now waiting for its getProcessOutput to finish + self.failUnlessEqual(len(s.pending_commands), 1) + self.failUnlessEqual(s.pending_commands[0][0], + ["info", "--xml", "--non-interactive", base]) + +def makeTime(timestring): + datefmt = '%Y/%m/%d %H:%M:%S' + when = time.mktime(time.strptime(timestring, datefmt)) + return when + + +class Everything(unittest.TestCase): + def test1(self): + s = MySvnSource(sample_base, split_file=split_file) + d = s.checksvn() + # the SvnSource is now waiting for its getProcessOutput to finish + self.failUnlessEqual(len(s.pending_commands), 1) + self.failUnlessEqual(s.pending_commands[0][0], + ["info", "--xml", "--non-interactive", + sample_base]) + d = s.pending_commands[0][1] + s.pending_commands.pop(0) + d.callback(sample_info_output) + # now it should be waiting for the 'svn log' command + self.failUnlessEqual(len(s.pending_commands), 1) + self.failUnlessEqual(s.pending_commands[0][0], + ["log", "--xml", "--verbose", "--non-interactive", + "--limit=100", sample_base]) + d = s.pending_commands[0][1] + s.pending_commands.pop(0) + d.callback(make_changes_output(1)) + # the command ignores the first batch of changes + self.failUnlessEqual(len(s.finished_changes), 0) + self.failUnlessEqual(s.last_change, 1) + + # now fire it again, nothing changing + d = s.checksvn() + self.failUnlessEqual(s.pending_commands[0][0], + ["log", "--xml", "--verbose", "--non-interactive", + "--limit=100", sample_base]) + d = s.pending_commands[0][1] + s.pending_commands.pop(0) + d.callback(make_changes_output(1)) + # nothing has changed + self.failUnlessEqual(len(s.finished_changes), 0) + self.failUnlessEqual(s.last_change, 1) + + # and again, with r2 this time + d = s.checksvn() + self.failUnlessEqual(s.pending_commands[0][0], + ["log", "--xml", "--verbose", "--non-interactive", + "--limit=100", sample_base]) + d = s.pending_commands[0][1] + s.pending_commands.pop(0) + d.callback(make_changes_output(2)) + # r2 should appear + self.failUnlessEqual(len(s.finished_changes), 1) + self.failUnlessEqual(s.last_change, 2) + + c = s.finished_changes[0] + self.failUnlessEqual(c.branch, "branch") + self.failUnlessEqual(c.revision, 2) + self.failUnlessEqual(c.files, ['']) + # TODO: this is what creating the branch looks like: a Change with a + # zero-length file. We should decide if we want filenames like this + # in the Change (and make sure nobody else gets confused by it) or if + # we want to strip them out. + self.failUnlessEqual(c.comments, "make_branch") + + # and again at r2, so nothing should change + d = s.checksvn() + self.failUnlessEqual(s.pending_commands[0][0], + ["log", "--xml", "--verbose", "--non-interactive", + "--limit=100", sample_base]) + d = s.pending_commands[0][1] + s.pending_commands.pop(0) + d.callback(make_changes_output(2)) + # nothing has changed + self.failUnlessEqual(len(s.finished_changes), 1) + self.failUnlessEqual(s.last_change, 2) + + # and again with both r3 and r4 appearing together + d = s.checksvn() + self.failUnlessEqual(s.pending_commands[0][0], + ["log", "--xml", "--verbose", "--non-interactive", + "--limit=100", sample_base]) + d = s.pending_commands[0][1] + s.pending_commands.pop(0) + d.callback(make_changes_output(4)) + self.failUnlessEqual(len(s.finished_changes), 3) + self.failUnlessEqual(s.last_change, 4) + + c3 = s.finished_changes[1] + self.failUnlessEqual(c3.branch, "branch") + self.failUnlessEqual(c3.revision, 3) + self.failUnlessEqual(c3.files, ["main.c"]) + self.failUnlessEqual(c3.comments, "commit_on_branch") + + c4 = s.finished_changes[2] + self.failUnlessEqual(c4.branch, None) + self.failUnlessEqual(c4.revision, 4) + self.failUnlessEqual(c4.files, ["version.c"]) + self.failUnlessEqual(c4.comments, "revised_to_2") + self.failUnlessEqual(c4.when, makeTime("2006/10/01 19:35:16")) + + +# TODO: +# get coverage of split_file returning None +# point at a live SVN server for a little while From warner at users.sourceforge.net Mon Oct 2 00:13:28 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:28 +0000 Subject: [Buildbot-commits] buildbot/buildbot/test test_svnpoller.py, 1.2, 1.3 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/test In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17160/buildbot/test Modified Files: test_svnpoller.py Log Message: [project @ svnpoller: clean up timestamps, unicode usage] Original author: warner at lothar.com Date: 2006-10-02 00:03:58 Index: test_svnpoller.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_svnpoller.py,v retrieving revision 1.2 retrieving revision 1.3 diff -u -d -r1.2 -r1.3 --- test_svnpoller.py 2 Oct 2006 00:13:18 -0000 1.2 +++ test_svnpoller.py 2 Oct 2006 00:13:26 -0000 1.3 @@ -421,7 +421,7 @@ self.failUnlessEqual(c4.revision, 4) self.failUnlessEqual(c4.files, ["version.c"]) self.failUnlessEqual(c4.comments, "revised_to_2") - self.failUnlessEqual(c4.when, makeTime("2006/10/01 19:35:16")) + self.failUnless(abs(c4.when - time.time()) < 60) # TODO: From warner at users.sourceforge.net Mon Oct 2 00:13:28 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:28 +0000 Subject: [Buildbot-commits] buildbot ChangeLog.svnpoller,1.1,1.2 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17160 Modified Files: ChangeLog.svnpoller Log Message: [project @ svnpoller: clean up timestamps, unicode usage] Original author: warner at lothar.com Date: 2006-10-02 00:03:58 Index: ChangeLog.svnpoller =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog.svnpoller,v retrieving revision 1.1 retrieving revision 1.2 diff -u -d -r1.1 -r1.2 --- ChangeLog.svnpoller 2 Oct 2006 00:13:11 -0000 1.1 +++ ChangeLog.svnpoller 2 Oct 2006 00:13:26 -0000 1.2 @@ -1,3 +1,18 @@ +Sun Oct 1 16:58:28 2006 Brian Warner + + * buildbot/changes/svnpoller.py (SvnSource.create_changes): + de-unicodify filenames before creating the Change, because the + rest of buildbot is unlikely to handle them well. Leave the 'who' + field as a unicode object.. I don't think there's anything that + will break very soon, and it will probably nudge us towards + accepting unicode everywhere sooner or later. Stop using the + "date" field that comes out of SVN, since it is using the + repository's clock (and timezone) and what we care about is the + buildmaster's (otherwise Changes from the future show up later + than the builds they triggered). + * buildbot/test/test_svnpoller.py (Everything.test1): match the + change to .when + Sun Oct 1 10:12:44 2006 Brian Warner * buildbot/changes/svnpoller.py (SvnSource): added Niklaus Giger's From warner at users.sourceforge.net Mon Oct 2 00:13:28 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:28 +0000 Subject: [Buildbot-commits] buildbot/buildbot/changes svnpoller.py,1.2,1.3 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/changes In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17160/buildbot/changes Modified Files: svnpoller.py Log Message: [project @ svnpoller: clean up timestamps, unicode usage] Original author: warner at lothar.com Date: 2006-10-02 00:03:58 Index: svnpoller.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/changes/svnpoller.py,v retrieving revision 1.2 retrieving revision 1.3 diff -u -d -r1.2 -r1.3 --- svnpoller.py 2 Oct 2006 00:13:18 -0000 1.2 +++ svnpoller.py 2 Oct 2006 00:13:26 -0000 1.3 @@ -392,15 +392,27 @@ # of that string up to b.s.source.SVN methods revision = int(el.getAttribute("revision")) dbgMsg("Adding change revision %s" % (revision,)) + # TODO: the rest of buildbot may not be ready for unicode 'who' + # values author = self._get_text(el, "author") comments = self._get_text(el, "msg") - when = self._get_text(el, "date") - when = time.mktime(time.strptime("%.19s" % when, - "%Y-%m-%dT%H:%M:%S")) + # there is a "date" field, but it provides localtime in the + # repository's timezone, whereas we care about buildmaster's + # localtime (since this will get used to position the boxes on + # the Waterfall display, etc). So ignore the date field and use + # our local clock instead. + #when = self._get_text(el, "date") + #when = time.mktime(time.strptime("%.19s" % when, + # "%Y-%m-%dT%H:%M:%S")) branches = {} pathlist = el.getElementsByTagName("paths")[0] for p in pathlist.getElementsByTagName("path"): path = "".join([t.data for t in p.childNodes]) + # the rest of buildbot is certaily not yet ready to handle + # unicode filenames, because they get put in RemoteCommands + # which get sent via PB to the buildslave, and PB doesn't + # handle unicode. + path = path.encode("ascii") if path.startswith("/"): path = path[1:] where = self._transform_path(path) @@ -417,7 +429,6 @@ files=branches[branch], comments=comments, revision=revision, - when=when, branch=branch) changes.append(c) From warner at users.sourceforge.net Mon Oct 2 00:13:34 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:34 +0000 Subject: [Buildbot-commits] buildbot ChangeLog.svnpoller,1.2,1.3 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17183 Modified Files: ChangeLog.svnpoller Log Message: [project @ html.py: improve unicode handling from zero to 'enough'] Original author: warner at lothar.com Date: 2006-10-02 00:04:42 Index: ChangeLog.svnpoller =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog.svnpoller,v retrieving revision 1.2 retrieving revision 1.3 diff -u -d -r1.2 -r1.3 --- ChangeLog.svnpoller 2 Oct 2006 00:13:26 -0000 1.2 +++ ChangeLog.svnpoller 2 Oct 2006 00:13:32 -0000 1.3 @@ -1,5 +1,10 @@ Sun Oct 1 16:58:28 2006 Brian Warner + * buildbot/status/html.py (HtmlResource.render): if we get a + unicode object from our content() method, encode it into utf-8 + like we've been claiming to all along. This allows the comments + and author names from svnpoller.py to be delivered properly. + * buildbot/changes/svnpoller.py (SvnSource.create_changes): de-unicodify filenames before creating the Change, because the rest of buildbot is unlikely to handle them well. Leave the 'who' From warner at users.sourceforge.net Mon Oct 2 00:13:34 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:34 +0000 Subject: [Buildbot-commits] buildbot/buildbot/status html.py,1.91,1.92 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/status In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17183/buildbot/status Modified Files: html.py Log Message: [project @ html.py: improve unicode handling from zero to 'enough'] Original author: warner at lothar.com Date: 2006-10-02 00:04:42 Index: html.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/status/html.py,v retrieving revision 1.91 retrieving revision 1.92 diff -u -d -r1.91 -r1.92 --- html.py 25 Sep 2006 02:43:56 -0000 1.91 +++ html.py 2 Oct 2006 00:13:32 -0000 1.92 @@ -144,14 +144,18 @@ class HtmlResource(Resource): css = None contentType = "text/html; charset=UTF-8" + title = "Dummy" + def render(self, request): data = self.content(request) + if isinstance(data, unicode): + data = data.encode("utf-8") request.setHeader("content-type", self.contentType) if request.method == "HEAD": request.setHeader("content-length", len(data)) return '' return data - title = "Dummy" + def content(self, request): data = ('\n" return data + def body(self, request): return "Dummy\n" From warner at users.sourceforge.net Mon Oct 2 00:13:42 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Mon, 02 Oct 2006 00:13:42 +0000 Subject: [Buildbot-commits] buildbot ChangeLog, 1.761, 1.762 ChangeLog.svnpoller, 1.3, NONE Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv17209 Modified Files: ChangeLog Removed Files: ChangeLog.svnpoller Log Message: [project @ svnpoller: merge ChangeLog entries] Original author: warner at lothar.com Date: 2006-10-02 00:08:59 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.761 retrieving revision 1.762 diff -u -d -r1.761 -r1.762 --- ChangeLog 30 Sep 2006 22:36:22 -0000 1.761 +++ ChangeLog 2 Oct 2006 00:13:40 -0000 1.762 @@ -1,3 +1,30 @@ +2006-10-01 Brian Warner + + * buildbot/status/html.py (HtmlResource.render): if we get a + unicode object from our content() method, encode it into utf-8 + like we've been claiming to all along. This allows the comments + and author names from svnpoller.py to be delivered properly. + + * buildbot/changes/svnpoller.py (SvnSource.create_changes): + de-unicodify filenames before creating the Change, because the + rest of buildbot is unlikely to handle them well. Leave the 'who' + field as a unicode object.. I don't think there's anything that + will break very soon, and it will probably nudge us towards + accepting unicode everywhere sooner or later. Stop using the + "date" field that comes out of SVN, since it is using the + repository's clock (and timezone) and what we care about is the + buildmaster's (otherwise Changes from the future show up later + than the builds they triggered). + * buildbot/test/test_svnpoller.py (Everything.test1): match the + change to .when + + * buildbot/changes/svnpoller.py (SvnSource): added Niklaus Giger's + Suvbersion repository polling ChangeSource. I've hacked it up + considerably: any bugs are entirely my own fault. Thank you + Niklaus! + * buildbot/test/test_svnpoller.py: tests for it + * docs/buildbot.texinfo (SvnSource): document it + 2006-09-30 Brian Warner * buildbot/scheduler.py (Periodic): submit a reason= to the --- ChangeLog.svnpoller DELETED --- From warner at users.sourceforge.net Tue Oct 3 17:23:16 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Tue, 03 Oct 2006 17:23:16 +0000 Subject: [Buildbot-commits] buildbot setup.py,1.40,1.41 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv22148 Modified Files: setup.py Log Message: [project @ setup.py: fix build/install errors involving unit test data files] Original author: warner at lothar.com Date: 2006-10-03 17:13:27 Index: setup.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/setup.py,v retrieving revision 1.40 retrieving revision 1.41 diff -u -d -r1.40 -r1.41 --- setup.py 25 Sep 2006 08:12:22 -0000 1.40 +++ setup.py 3 Oct 2006 17:23:14 -0000 1.41 @@ -41,7 +41,8 @@ for f in os.listdir("buildbot/test/mail"): if f.endswith("~"): continue - testmsgs.append("buildbot/test/mail/%s" % f) + if f.startswith("msg") or f.startswith("syncmail"): + testmsgs.append("buildbot/test/mail/%s" % f) setup(name="buildbot", version=version, @@ -71,13 +72,13 @@ "buildbot.slave", "buildbot.scripts", "buildbot.test", - "buildbot.test.subdir", ], data_files=[("buildbot", ["buildbot/buildbot.png"]), ("buildbot/clients", ["buildbot/clients/debug.glade"]), ("buildbot/status", ["buildbot/status/classic.css"]), ("buildbot/scripts", ["buildbot/scripts/sample.cfg"]), ("buildbot/test/mail", testmsgs), + ("buildbot/test/subdir", ["buildbot/test/subdir/emit.py"]), ], scripts = scripts, cmdclass={'install_data': install_data_twisted}, From warner at users.sourceforge.net Thu Oct 5 00:45:50 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Thu, 05 Oct 2006 00:45:50 +0000 Subject: [Buildbot-commits] buildbot/buildbot/steps python.py,1.4,1.5 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/steps In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv25037/buildbot/steps Modified Files: python.py Log Message: [project @ PyFlakes: ignore initial output lines that weren't emitted by pyflakes] Original author: warner at allmydata.com Date: 2006-10-05 00:39:11 Index: python.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/steps/python.py,v retrieving revision 1.4 retrieving revision 1.5 diff -u -d -r1.4 -r1.5 --- python.py 20 Sep 2006 06:33:35 -0000 1.4 +++ python.py 5 Oct 2006 00:45:48 -0000 1.5 @@ -65,7 +65,19 @@ counts[m] = 0 summaries[m] = [] + first = True for line in StringIO(log.getText()).readlines(): + # the first few lines might contain echoed commands from a 'make + # pyflakes' step, so don't count these as warnings. Stop ignoring + # the initial lines as soon as we see one with a colon. + if first: + if line.find(":") != -1: + # there's the colon, this is the first real line + first = False + # fall through and parse the line + else: + # skip this line, keep skipping non-colon lines + continue if line.find("imported but unused") != -1: m = "unused" elif line.find("*' used; unable to detect undefined names") != -1: From warner at users.sourceforge.net Thu Oct 5 00:45:50 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Thu, 05 Oct 2006 00:45:50 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.762,1.763 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv25037 Modified Files: ChangeLog Log Message: [project @ PyFlakes: ignore initial output lines that weren't emitted by pyflakes] Original author: warner at allmydata.com Date: 2006-10-05 00:39:11 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.762 retrieving revision 1.763 diff -u -d -r1.762 -r1.763 --- ChangeLog 2 Oct 2006 00:13:40 -0000 1.762 +++ ChangeLog 5 Oct 2006 00:45:47 -0000 1.763 @@ -1,3 +1,11 @@ +2006-10-04 Brian Warner + + * buildbot/steps/python.py (PyFlakes.createSummary): skip any + initial lines that weren't emitted by pyflakes. When the pyflakes + command is run under a Makefile, 'make' will echo the command it + runs to stdio, and that was getting logged as a "misc" warning. + * buildbot/test/test_steps.py (Python.testPyFlakes2): test it + 2006-10-01 Brian Warner * buildbot/status/html.py (HtmlResource.render): if we get a From warner at users.sourceforge.net Thu Oct 5 00:45:50 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Thu, 05 Oct 2006 00:45:50 +0000 Subject: [Buildbot-commits] buildbot/buildbot/test test_steps.py,1.31,1.32 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/test In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv25037/buildbot/test Modified Files: test_steps.py Log Message: [project @ PyFlakes: ignore initial output lines that weren't emitted by pyflakes] Original author: warner at allmydata.com Date: 2006-10-05 00:39:11 Index: test_steps.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_steps.py,v retrieving revision 1.31 retrieving revision 1.32 diff -u -d -r1.31 -r1.32 --- test_steps.py 19 Sep 2006 18:46:55 -0000 1.31 +++ test_steps.py 5 Oct 2006 00:45:48 -0000 1.32 @@ -394,11 +394,12 @@ return maybeWait(d) class Python(StepTester, unittest.TestCase): - def testPyFlakes(self): - self.masterbase = "Python.master" + def testPyFlakes1(self): + self.masterbase = "Python.testPyFlakes1" step = self.makeStep(python.PyFlakes) output = \ -"""buildbot/changes/freshcvsmail.py:5: 'FCMaildirSource' imported but unused +"""pyflakes buildbot +buildbot/changes/freshcvsmail.py:5: 'FCMaildirSource' imported but unused buildbot/clients/debug.py:9: redefinition of unused 'gtk' from line 9 buildbot/clients/debug.py:9: 'gnome' imported but unused buildbot/scripts/runner.py:323: redefinition of unused 'run' from line 321 @@ -415,6 +416,7 @@ self.failUnless("undefined=1" in desc) self.failUnless("redefs=3" in desc) self.failUnless("import*=1" in desc) + self.failIf("misc=" in desc) self.failUnlessEqual(step.getProperty("pyflakes-unused"), 2) self.failUnlessEqual(step.getProperty("pyflakes-undefined"), 1) @@ -438,3 +440,31 @@ cmd.rc = 0 results = step.evaluateCommand(cmd) self.failUnlessEqual(results, FAILURE) # because of the 'undefined' + + def testPyFlakes2(self): + self.masterbase = "Python.testPyFlakes2" + step = self.makeStep(python.PyFlakes) + output = \ +"""pyflakes buildbot +some more text here that should be ignored +buildbot/changes/freshcvsmail.py:5: 'FCMaildirSource' imported but unused +buildbot/clients/debug.py:9: redefinition of unused 'gtk' from line 9 +buildbot/clients/debug.py:9: 'gnome' imported but unused +buildbot/scripts/runner.py:323: redefinition of unused 'run' from line 321 +buildbot/scripts/runner.py:325: redefinition of unused 'run' from line 323 +buildbot/scripts/imaginary.py:12: undefined name 'size' +could not compile 'blah/blah.py':3: +pretend there was an invalid line here +buildbot/scripts/imaginary.py:18: 'from buildbot import *' used; unable to detect undefined names +""" + log = step.addLog("stdio") + log.addStdout(output) + log.finish() + step.createSummary(log) + desc = step.descriptionDone + self.failUnless("unused=2" in desc) + self.failUnless("undefined=1" in desc) + self.failUnless("redefs=3" in desc) + self.failUnless("import*=1" in desc) + self.failUnless("misc=2" in desc) + From warner at users.sourceforge.net Thu Oct 5 00:46:00 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Thu, 05 Oct 2006 00:46:00 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.763,1.764 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv25060 Modified Files: ChangeLog Log Message: [project @ PyFlakes: another test] Original author: warner at allmydata.com Date: 2006-10-05 00:42:17 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.763 retrieving revision 1.764 diff -u -d -r1.763 -r1.764 --- ChangeLog 5 Oct 2006 00:45:47 -0000 1.763 +++ ChangeLog 5 Oct 2006 00:45:57 -0000 1.764 @@ -5,6 +5,7 @@ command is run under a Makefile, 'make' will echo the command it runs to stdio, and that was getting logged as a "misc" warning. * buildbot/test/test_steps.py (Python.testPyFlakes2): test it + * buildbot/test/test_steps.py (testPyFlakes3): another test 2006-10-01 Brian Warner From warner at users.sourceforge.net Thu Oct 5 00:46:00 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Thu, 05 Oct 2006 00:46:00 +0000 Subject: [Buildbot-commits] buildbot/buildbot/test test_steps.py,1.32,1.33 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/test In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv25060/buildbot/test Modified Files: test_steps.py Log Message: [project @ PyFlakes: another test] Original author: warner at allmydata.com Date: 2006-10-05 00:42:17 Index: test_steps.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_steps.py,v retrieving revision 1.32 retrieving revision 1.33 diff -u -d -r1.32 -r1.33 --- test_steps.py 5 Oct 2006 00:45:48 -0000 1.32 +++ test_steps.py 5 Oct 2006 00:45:58 -0000 1.33 @@ -468,3 +468,27 @@ self.failUnless("import*=1" in desc) self.failUnless("misc=2" in desc) + + def testPyFlakes3(self): + self.masterbase = "Python.testPyFlakes3" + step = self.makeStep(python.PyFlakes) + output = \ +"""buildbot/changes/freshcvsmail.py:5: 'FCMaildirSource' imported but unused +buildbot/clients/debug.py:9: redefinition of unused 'gtk' from line 9 +buildbot/clients/debug.py:9: 'gnome' imported but unused +buildbot/scripts/runner.py:323: redefinition of unused 'run' from line 321 +buildbot/scripts/runner.py:325: redefinition of unused 'run' from line 323 +buildbot/scripts/imaginary.py:12: undefined name 'size' +buildbot/scripts/imaginary.py:18: 'from buildbot import *' used; unable to detect undefined names +""" + log = step.addLog("stdio") + log.addStdout(output) + log.finish() + step.createSummary(log) + desc = step.descriptionDone + self.failUnless("unused=2" in desc) + self.failUnless("undefined=1" in desc) + self.failUnless("redefs=3" in desc) + self.failUnless("import*=1" in desc) + self.failIf("misc" in desc) + From warner at users.sourceforge.net Sun Oct 8 20:54:19 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 08 Oct 2006 20:54:19 +0000 Subject: [Buildbot-commits] buildbot/contrib svn_watcher.py,1.1,1.2 Message-ID: Update of /cvsroot/buildbot/buildbot/contrib In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6339/contrib Modified Files: svn_watcher.py Log Message: [project @ security fix to contrib/svn_watcher.py] Original author: warner at lothar.com Date: 2006-10-08 20:48:35 Index: svn_watcher.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/contrib/svn_watcher.py,v retrieving revision 1.1 retrieving revision 1.2 diff -u -d -r1.1 -r1.2 --- svn_watcher.py 17 Apr 2006 18:11:42 -0000 1.1 +++ svn_watcher.py 8 Oct 2006 20:54:17 -0000 1.2 @@ -16,7 +16,7 @@ # 15.03.06 by John Pye # 29.03.06 by Niklaus Giger, added support to run under windows, added invocation option -import commands +import subprocess import xml.dom.minidom import sys import time @@ -24,8 +24,13 @@ if sys.platform == 'win32': import win32pipe +def getoutput(cmd): + p = subprocess.Popen(cmd, stdout=subprocess.PIPE) + return p.stdout.read() + def checkChanges(repo, master, verbose=False, oldRevision=-1): - cmd ="svn log --non-interactive --xml --verbose --limit=1 "+repo + cmd = ["svn", "log", "--non-interactive", "--xml", "--verbose", + "--limit=1", repo] if verbose == True: print "Getting last revision of repository: " + repo @@ -34,11 +39,11 @@ xml1 = ''.join(f.readlines()) f.close() else: - xml1 = commands.getoutput(cmd) - + xml1 = getoutput(cmd) + if verbose == True: print "XML\n-----------\n"+xml1+"\n\n" - + doc = xml.dom.minidom.parseString(xml1) el = doc.getElementsByTagName("logentry")[0] revision = el.getAttribute("revision") @@ -57,20 +62,23 @@ print paths if revision != oldRevision: - cmd = "buildbot sendchange --master="+master+" --revision=\""+revision+"\" --username=\""+author+"\"--comments=\""+comments+"\" "+" ".join(paths) - + cmd = ["buildbot", "sendchange", "--master=%s"%master, + "--revision=%s"%revision, "--username=%s"%author, + "--comments=%s"%comments] + cmd += paths + if verbose == True: print cmd - + if sys.platform == 'win32': f = win32pipe.popen(cmd) print time.strftime("%H.%M.%S ") + "Revision "+revision+ ": "+ ''.join(f.readlines()) f.close() else: - xml1 = commands.getoutput(cmd) + xml1 = getoutput(cmd) else: print time.strftime("%H.%M.%S ") + "nothing has changed since revision "+revision - + return revision if __name__ == '__main__': From warner at users.sourceforge.net Sun Oct 8 20:54:19 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 08 Oct 2006 20:54:19 +0000 Subject: [Buildbot-commits] buildbot CREDITS,1.4,1.5 ChangeLog,1.764,1.765 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6339 Modified Files: CREDITS ChangeLog Log Message: [project @ security fix to contrib/svn_watcher.py] Original author: warner at lothar.com Date: 2006-10-08 20:48:35 Index: CREDITS =================================================================== RCS file: /cvsroot/buildbot/buildbot/CREDITS,v retrieving revision 1.4 retrieving revision 1.5 diff -u -d -r1.4 -r1.5 --- CREDITS 17 Sep 2006 20:49:31 -0000 1.4 +++ CREDITS 8 Oct 2006 20:54:16 -0000 1.5 @@ -44,3 +44,4 @@ Albert Hofkamp Brett Neely Wade Brainerd +Nick Mathewson Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.764 retrieving revision 1.765 diff -u -d -r1.764 -r1.765 --- ChangeLog 5 Oct 2006 00:45:57 -0000 1.764 +++ ChangeLog 8 Oct 2006 20:54:16 -0000 1.765 @@ -1,3 +1,12 @@ +2006-10-08 Brian Warner + + * contrib/svn_watcher.py: fix security holes by using proper argv + arrays and subprocess.Popen() rather than commands.getoutput(). + Thanks to Nick Mathewson for the patch. Note that svn_watcher.py + is deprecated in favor of buildbot/changes/svnpoller.py, and will + probably be removed by the next release. + * CREDITS: add Nick + 2006-10-04 Brian Warner * buildbot/steps/python.py (PyFlakes.createSummary): skip any From warner at users.sourceforge.net Fri Oct 13 07:30:03 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 13 Oct 2006 07:30:03 +0000 Subject: [Buildbot-commits] buildbot/buildbot/steps transfer.py,1.3,1.4 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/steps In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6047/buildbot/steps Modified Files: transfer.py Log Message: [project @ filetransfer: code cleanups] Original author: warner at lothar.com Date: 2006-10-13 07:28:48 Index: transfer.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/steps/transfer.py,v retrieving revision 1.3 retrieving revision 1.4 diff -u -d -r1.3 -r1.4 --- transfer.py 17 Sep 2006 20:35:49 -0000 1.3 +++ transfer.py 13 Oct 2006 07:29:58 -0000 1.4 @@ -104,20 +104,20 @@ name = 'upload' - def __init__(self, build, **kwargs): - buildstep_kwargs = {} - for k in kwargs.keys()[:]: - if k in BuildStep.parms: - buildstep_kwargs[k] = kwargs[k] - del kwargs[k] - BuildStep.__init__(self, build, **buildstep_kwargs) + def __init__(self, build, slavesrc, masterdest, + workdir="build", maxsize=None, blocksize=16*1024, + **buildstep_kwargs): + BuildStep.__init__(self, build, **buildstep_kwargs) - self.args = kwargs - self.fileWriter = None + self.slavesrc = slavesrc + self.masterdest = masterdest + self.workdir = workdir + self.maxsize = maxsize + self.blocksize = blocksize def start(self): - source = self.args['slavesrc'] - masterdest = self.args['masterdest'] + source = self.slavesrc + masterdest = self.masterdest # we rely upon the fact that the buildmaster runs chdir'ed into its # basedir to make sure that relative paths in masterdest are expanded # properly. TODO: maybe pass the master's basedir all the way down @@ -134,12 +134,12 @@ # default arguments args = { - 'maxsize': None, - 'blocksize': 16*1024, - 'workdir': 'build', + 'slavesrc': source, + 'workdir': self.workdir, + 'writer': self.fileWriter, + 'maxsize': self.maxsize, + 'blocksize': self.blocksize } - args.update(self.args) - args['writer'] = self.fileWriter self.cmd = StatusRemoteCommand('uploadFile', args) d = self.runCommand(self.cmd) @@ -173,20 +173,20 @@ name = 'download' - def __init__(self,build, **kwargs): - buildstep_kwargs = {} - for k in kwargs.keys()[:]: - if k in BuildStep.parms: - buildstep_kwargs[k] = kwargs[k] - del kwargs[k] - BuildStep.__init__(self, build, **buildstep_kwargs) + def __init__(self, build, mastersrc, slavedest, + workdir="build", maxsize=None, blocksize=16*1024, + **buildstep_kwargs): + BuildStep.__init__(self, build, **buildstep_kwargs) - self.args = kwargs - self.fileReader = None + self.mastersrc = mastersrc + self.slavedest = slavedest + self.workdir = workdir + self.maxsize = maxsize + self.blocksize = blocksize def start(self): - source = os.path.expanduser(self.args['mastersrc']) - slavedest = self.args['slavedest'] + source = os.path.expanduser(self.mastersrc) + slavedest = self.slavedest log.msg("FileDownload started, from master %r to slave %r" % (source, slavedest)) @@ -206,12 +206,12 @@ # default arguments args = { - 'maxsize': None, - 'blocksize': 16*1024, - 'workdir': 'build', + 'slavedest': self.slavedest, + 'maxsize': self.maxsize, + 'reader': self.fileReader, + 'blocksize': self.blocksize, + 'workdir': self.workdir, } - args.update(self.args) - args['reader'] = self.fileReader self.cmd = StatusRemoteCommand('downloadFile', args) d = self.runCommand(self.cmd) From warner at users.sourceforge.net Fri Oct 13 07:30:00 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 13 Oct 2006 07:30:00 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.765,1.766 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6047 Modified Files: ChangeLog Log Message: [project @ filetransfer: code cleanups] Original author: warner at lothar.com Date: 2006-10-13 07:28:48 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.765 retrieving revision 1.766 diff -u -d -r1.765 -r1.766 --- ChangeLog 8 Oct 2006 20:54:16 -0000 1.765 +++ ChangeLog 13 Oct 2006 07:29:57 -0000 1.766 @@ -1,3 +1,12 @@ +2006-10-12 Brian Warner + + * buildbot/steps/transfer.py: rework __init__ and args setup + * buildbot/slave/commands.py (SlaveFileDownloadCommand): minor + docs improvements + * buildbot/slave/commands.py (SlaveFileDownloadCommand.setup): + when opening the target file, only catch IOError (to report via + stderr/rc!=0), let the others be reported as normal exceptions + 2006-10-08 Brian Warner * contrib/svn_watcher.py: fix security holes by using proper argv From warner at users.sourceforge.net Fri Oct 13 07:30:00 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 13 Oct 2006 07:30:00 +0000 Subject: [Buildbot-commits] buildbot/buildbot/slave commands.py,1.68,1.69 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/slave In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6047/buildbot/slave Modified Files: commands.py Log Message: [project @ filetransfer: code cleanups] Original author: warner at lothar.com Date: 2006-10-13 07:28:48 Index: commands.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/slave/commands.py,v retrieving revision 1.68 retrieving revision 1.69 diff -u -d -r1.68 -r1.69 --- commands.py 30 Sep 2006 19:32:48 -0000 1.68 +++ commands.py 13 Oct 2006 07:29:58 -0000 1.69 @@ -675,14 +675,12 @@ Upload a file from slave to build master Arguments: - - ['workdir']: directory to use - - ['slavesrc']: name of the file to upload to the buildmaster - - ['writer']: object for remote writing + - ['workdir']: base directory to use + - ['slavesrc']: name of the slave-side file to read from + - ['writer']: RemoteReference to a transfer._FileWriter object - ['maxsize']: max size (in bytes) of file to write - - ['blocksize']: max size for one data block - + - ['blocksize']: max size for each data block """ - debug = False def setup(self,args): @@ -786,12 +784,11 @@ Download a file from master to slave Arguments: - - ['workdir']: directory to use - - ['slavedest']: name of the file to upload to the buildmaster - - ['reader']: object for remote reading + - ['workdir']: base directory to use + - ['slavedest']: name of the slave-side file to be created + - ['reader']: RemoteReference to a transfer._FileReader object - ['maxsize']: max size (in bytes) of file to write - - ['blocksize']: max size for one data block - + - ['blocksize']: max size for each data block """ debug = False @@ -815,7 +812,7 @@ self.fp = open(self.path, 'w') if self.debug: log.msg('Opened %r for download' % self.path) - except: + except IOError: self.fp = None self.stderr = 'Cannot open file %r for download' % self.path self.rc = 1 From warner at users.sourceforge.net Fri Oct 13 09:08:27 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 13 Oct 2006 09:08:27 +0000 Subject: [Buildbot-commits] buildbot/buildbot/steps transfer.py,1.4,1.5 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/steps In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv12999/buildbot/steps Modified Files: transfer.py Log Message: [project @ transfer.py: fix a stupid error] Original author: warner at lothar.com Date: 2006-10-13 09:06:53 Index: transfer.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/steps/transfer.py,v retrieving revision 1.4 retrieving revision 1.5 diff -u -d -r1.4 -r1.5 --- transfer.py 13 Oct 2006 07:29:58 -0000 1.4 +++ transfer.py 13 Oct 2006 09:08:25 -0000 1.5 @@ -129,7 +129,7 @@ self.step_status.setColor('yellow') self.step_status.setText(['uploading', source]) - fp = open(self.args['masterdest'],'w') + fp = open(self.masterdest, 'w') self.fileWriter = _FileWriter(fp) # default arguments From warner at users.sourceforge.net Fri Oct 13 09:08:27 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 13 Oct 2006 09:08:27 +0000 Subject: [Buildbot-commits] buildbot/buildbot/slave bot.py,1.20,1.21 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/slave In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv12999/buildbot/slave Modified Files: bot.py Log Message: [project @ transfer.py: fix a stupid error] Original author: warner at lothar.com Date: 2006-10-13 09:06:53 Index: bot.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/slave/bot.py,v retrieving revision 1.20 retrieving revision 1.21 diff -u -d -r1.20 -r1.21 --- bot.py 6 Sep 2006 00:41:55 -0000 1.20 +++ bot.py 13 Oct 2006 09:08:25 -0000 1.21 @@ -65,7 +65,7 @@ #service.Service.__init__(self) # Service has no __init__ method self.setName(name) self.not_really = not_really - + def __repr__(self): return "" % self.name From warner at users.sourceforge.net Fri Oct 13 09:08:27 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 13 Oct 2006 09:08:27 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.766,1.767 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv12999 Modified Files: ChangeLog Log Message: [project @ transfer.py: fix a stupid error] Original author: warner at lothar.com Date: 2006-10-13 09:06:53 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.766 retrieving revision 1.767 diff -u -d -r1.766 -r1.767 --- ChangeLog 13 Oct 2006 07:29:57 -0000 1.766 +++ ChangeLog 13 Oct 2006 09:08:25 -0000 1.767 @@ -1,3 +1,9 @@ +2006-10-13 Brian Warner + + * buildbot/steps/transfer.py (FileUpload.start): Fix stupid error. + Maybe I should run my own unit tests before recording a big + change. Good thing I've got a buildbot to remind me. + 2006-10-12 Brian Warner * buildbot/steps/transfer.py: rework __init__ and args setup From warner at users.sourceforge.net Fri Oct 13 09:14:09 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 13 Oct 2006 09:14:09 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.767,1.768 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv15346 Modified Files: ChangeLog Log Message: [project @ docs: update some examples to match reorganized directories] Original author: warner at lothar.com Date: 2006-10-13 09:10:06 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.767 retrieving revision 1.768 diff -u -d -r1.767 -r1.768 --- ChangeLog 13 Oct 2006 09:08:25 -0000 1.767 +++ ChangeLog 13 Oct 2006 09:14:07 -0000 1.768 @@ -1,5 +1,8 @@ 2006-10-13 Brian Warner + * docs/buildbot.texinfo (Adding LogObservers): update sample code + to match the great Steps renaming + * buildbot/steps/transfer.py (FileUpload.start): Fix stupid error. Maybe I should run my own unit tests before recording a big change. Good thing I've got a buildbot to remind me. From warner at users.sourceforge.net Fri Oct 13 09:14:09 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 13 Oct 2006 09:14:09 +0000 Subject: [Buildbot-commits] buildbot/docs buildbot.texinfo,1.83,1.84 Message-ID: Update of /cvsroot/buildbot/buildbot/docs In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv15346/docs Modified Files: buildbot.texinfo Log Message: [project @ docs: update some examples to match reorganized directories] Original author: warner at lothar.com Date: 2006-10-13 09:10:06 Index: buildbot.texinfo =================================================================== RCS file: /cvsroot/buildbot/buildbot/docs/buildbot.texinfo,v retrieving revision 1.83 retrieving revision 1.84 diff -u -d -r1.83 -r1.84 --- buildbot.texinfo 2 Oct 2006 00:13:18 -0000 1.83 +++ buildbot.texinfo 13 Oct 2006 09:14:07 -0000 1.84 @@ -4274,10 +4274,11 @@ progress that this event represents. There are a number of pre-built @code{LogObserver} classes that you -can choose from, and of course you can subclass them to add further -customization. The @code{LogLineObserver} class handles the grunt work -of buffering and scanning for end-of-line delimiters, allowing your -parser to operate on complete stdout/stderr lines. +can choose from (defined in @code{buildbot.process.buildstep}, and of +course you can subclass them to add further customization. The + at code{LogLineObserver} class handles the grunt work of buffering and +scanning for end-of-line delimiters, allowing your parser to operate +on complete stdout/stderr lines. For example, let's take a look at the @code{TrialTestCaseCounter}, which is used by the Trial step to count test cases as they are run. @@ -4298,7 +4299,9 @@ The parser class looks like this: @example -class TrialTestCaseCounter(step.LogLineObserver): +from buildbot.process.buildstep import LogLineObserver + +class TrialTestCaseCounter(LogLineObserver): _line_re = re.compile(r'^([\w\.]+) \.\.\. \[([^\]]+)\]$') numTests = 0 finished = False From warner at users.sourceforge.net Sun Oct 15 17:51:13 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 15 Oct 2006 17:51:13 +0000 Subject: [Buildbot-commits] buildbot/buildbot interfaces.py,1.47,1.48 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv10284/buildbot Modified Files: interfaces.py Log Message: [project @ IStatus.getURLForThing was misspelled in the interface definition] Original author: warner at lothar.com Date: 2006-10-15 17:34:29 Index: interfaces.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/interfaces.py,v retrieving revision 1.47 retrieving revision 1.48 diff -u -d -r1.47 -r1.48 --- interfaces.py 15 Sep 2006 14:48:53 -0000 1.47 +++ interfaces.py 15 Oct 2006 17:51:11 -0000 1.48 @@ -98,7 +98,7 @@ def getBuildbotURL(): """Return the URL of the top-most Buildbot status page, or None if this Buildbot does not provide a web status page.""" - def getURLFor(thing): + def getURLForThing(thing): """Return the URL of a page which provides information on 'thing', which should be an object that implements one of the status interfaces defined in L{buildbot.interfaces}. Returns None if no From warner at users.sourceforge.net Sun Oct 15 17:51:13 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 15 Oct 2006 17:51:13 +0000 Subject: [Buildbot-commits] buildbot CREDITS,1.5,1.6 ChangeLog,1.768,1.769 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv10284 Modified Files: CREDITS ChangeLog Log Message: [project @ IStatus.getURLForThing was misspelled in the interface definition] Original author: warner at lothar.com Date: 2006-10-15 17:34:29 Index: CREDITS =================================================================== RCS file: /cvsroot/buildbot/buildbot/CREDITS,v retrieving revision 1.5 retrieving revision 1.6 diff -u -d -r1.5 -r1.6 --- CREDITS 8 Oct 2006 20:54:16 -0000 1.5 +++ CREDITS 15 Oct 2006 17:51:11 -0000 1.6 @@ -45,3 +45,4 @@ Brett Neely Wade Brainerd Nick Mathewson +Roy Rapoport Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.768 retrieving revision 1.769 diff -u -d -r1.768 -r1.769 --- ChangeLog 13 Oct 2006 09:14:07 -0000 1.768 +++ ChangeLog 15 Oct 2006 17:51:11 -0000 1.769 @@ -1,3 +1,9 @@ +2006-10-15 Brian Warner + + * buildbot/interfaces.py (IStatus.getURLForThing): oops, the + method name was misspelled in the interface definition. Thanks to + Roy Rapoport for the catch. + 2006-10-13 Brian Warner * docs/buildbot.texinfo (Adding LogObservers): update sample code