From warner at users.sourceforge.net Sun Nov 5 04:56:05 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 05 Nov 2006 04:56:05 +0000 Subject: [Buildbot-commits] buildbot/buildbot/slave commands.py,1.69,1.70 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/slave In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv28649/buildbot/slave Modified Files: commands.py Log Message: [project @ Seek to current file position to clear EOF status on Mac OS X.] Mac OS X and Linux differ in behaviour when reading from a file that has previously reached EOF. On Linux, any new data that has been appended to the file will be returned. On Mac OS X, the empty string will always be returned. Seeking to the current position in the file resets the EOF flag on Mac OS X and will allow future reads to work as intended. Original author: code at bdash.net.nz Date: 2006-11-02 02:55:04 Index: commands.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/slave/commands.py,v retrieving revision 1.69 retrieving revision 1.70 diff -u -d -r1.69 -r1.70 --- commands.py 13 Oct 2006 07:29:58 -0000 1.69 +++ commands.py 5 Nov 2006 04:56:03 -0000 1.70 @@ -196,6 +196,7 @@ return # no file to work with self.f = open(self.logfile, "rb") self.started = True + self.f.seek(self.f.tell(), 0) while True: data = self.f.read(10000) if not data: From warner at users.sourceforge.net Sun Nov 5 04:56:13 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 05 Nov 2006 04:56:13 +0000 Subject: [Buildbot-commits] buildbot CREDITS,1.6,1.7 ChangeLog,1.769,1.770 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv28886 Modified Files: CREDITS ChangeLog Log Message: [project @ add changelog/credit for Mark Rowe's OS-X LogFileWatcher patch] Original author: warner at lothar.com Date: 2006-11-05 04:55:16 Index: CREDITS =================================================================== RCS file: /cvsroot/buildbot/buildbot/CREDITS,v retrieving revision 1.6 retrieving revision 1.7 diff -u -d -r1.6 -r1.7 --- CREDITS 15 Oct 2006 17:51:11 -0000 1.6 +++ CREDITS 5 Nov 2006 04:56:11 -0000 1.7 @@ -46,3 +46,4 @@ Wade Brainerd Nick Mathewson Roy Rapoport +Mark Rowe Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.769 retrieving revision 1.770 diff -u -d -r1.769 -r1.770 --- ChangeLog 15 Oct 2006 17:51:11 -0000 1.769 +++ ChangeLog 5 Nov 2006 04:56:11 -0000 1.770 @@ -1,3 +1,10 @@ +2006-11-04 Brian Warner + + * buildbot/slave/commands.py (LogFileWatcher.poll): overcome a + linux-vs-osx behavior difference w.r.t. reading from files that + have reached EOF. This should fix LogFileWatcher on OS-X. Thanks + to Mark Rowe for the patch. + 2006-10-15 Brian Warner * buildbot/interfaces.py (IStatus.getURLForThing): oops, the From warner at users.sourceforge.net Sun Nov 5 05:13:08 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 05 Nov 2006 05:13:08 +0000 Subject: [Buildbot-commits] buildbot/buildbot/status tinderbox.py,1.2,1.3 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/status In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv3138/buildbot/status Modified Files: tinderbox.py Log Message: [project @ apply patch by Dave Liebreich to update the tinderbox status mailer: closes SF#1587352] Original author: warner at lothar.com Date: 2006-11-05 05:04:36 Index: tinderbox.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/status/tinderbox.py,v retrieving revision 1.2 retrieving revision 1.3 diff -u -d -r1.2 -r1.3 --- tinderbox.py 29 Sep 2006 07:04:21 -0000 1.2 +++ tinderbox.py 5 Nov 2006 05:13:06 -0000 1.3 @@ -115,9 +115,12 @@ if results == "building": res = "building" text += res - elif results == WARNINGS or results == SUCCESS: + elif results == SUCCESS: res = "success" text += res + elif results == WARNINGS: + res = "testfailed" + text += res else: res += "busted" text += res From warner at users.sourceforge.net Sun Nov 5 05:13:08 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 05 Nov 2006 05:13:08 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.770,1.771 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv3138 Modified Files: ChangeLog Log Message: [project @ apply patch by Dave Liebreich to update the tinderbox status mailer: closes SF#1587352] Original author: warner at lothar.com Date: 2006-11-05 05:04:36 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.770 retrieving revision 1.771 diff -u -d -r1.770 -r1.771 --- ChangeLog 5 Nov 2006 04:56:11 -0000 1.770 +++ ChangeLog 5 Nov 2006 05:13:06 -0000 1.771 @@ -1,5 +1,10 @@ 2006-11-04 Brian Warner + * buildbot/status/tinderbox.py + (TinderboxMailNotifier.buildMessage): send out a "testfailed" + status when the build results in WARNINGS. Patch from Dave + Liebreich. Closes SF#1587352. + * buildbot/slave/commands.py (LogFileWatcher.poll): overcome a linux-vs-osx behavior difference w.r.t. reading from files that have reached EOF. This should fix LogFileWatcher on OS-X. Thanks From warner at users.sourceforge.net Sun Nov 5 05:13:16 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 05 Nov 2006 05:13:16 +0000 Subject: [Buildbot-commits] buildbot CREDITS,1.7,1.8 ChangeLog,1.771,1.772 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv3175 Modified Files: CREDITS ChangeLog Log Message: [project @ bonsaipoller: apply updates+test from Ben Hearsum. Closes SF#1590310.] Original author: warner at lothar.com Date: 2006-11-05 05:11:54 Index: CREDITS =================================================================== RCS file: /cvsroot/buildbot/buildbot/CREDITS,v retrieving revision 1.7 retrieving revision 1.8 diff -u -d -r1.7 -r1.8 --- CREDITS 5 Nov 2006 04:56:11 -0000 1.7 +++ CREDITS 5 Nov 2006 05:13:14 -0000 1.8 @@ -47,3 +47,5 @@ Nick Mathewson Roy Rapoport Mark Rowe +Ben Hearsum +Dave Liebreich Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.771 retrieving revision 1.772 diff -u -d -r1.771 -r1.772 --- ChangeLog 5 Nov 2006 05:13:06 -0000 1.771 +++ ChangeLog 5 Nov 2006 05:13:14 -0000 1.772 @@ -1,5 +1,9 @@ 2006-11-04 Brian Warner + * buildbot/changes/bonsaipoller.py: apply updates from Ben + Hearsum. Closes SF#1590310. + * buildbot/test/test_bonsaipoller.py: and tests + * buildbot/status/tinderbox.py (TinderboxMailNotifier.buildMessage): send out a "testfailed" status when the build results in WARNINGS. Patch from Dave From warner at users.sourceforge.net Sun Nov 5 05:13:16 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 05 Nov 2006 05:13:16 +0000 Subject: [Buildbot-commits] buildbot/buildbot/test test_bonsaipoller.py, NONE, 1.1 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/test In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv3175/buildbot/test Added Files: test_bonsaipoller.py Log Message: [project @ bonsaipoller: apply updates+test from Ben Hearsum. Closes SF#1590310.] Original author: warner at lothar.com Date: 2006-11-05 05:11:54 --- NEW FILE: test_bonsaipoller.py --- # -*- test-case-name: buildbot.test.test_bonsaipoller -*- from twisted.trial import unittest from buildbot.changes.bonsaipoller import * from StringIO import StringIO from copy import deepcopy import re log1 = "Add Bug 338541a" who1 = "sar at gmail.com" date1 = 1161908700 log2 = "bug 357427 add static ctor/dtor methods" who2 = "aarrg at ooacm.org" date2 = 1161910620 log3 = "Testing log #3 lbah blah" who3 = "huoents at hueont.net" date3 = 1889822728 rev1 = "1.8" file1 = "mozilla/testing/mochitest/tests/index.html" rev2 = "1.1" file2 = "mozilla/testing/mochitest/tests/test_bug338541.xhtml" rev3 = "1.1812" file3 = "mozilla/xpcom/threads/nsAutoLock.cpp" rev4 = "1.3" file4 = "mozilla/xpcom/threads/nsAutoLock.h" rev5 = "2.4" file5 = "mozilla/xpcom/threads/test.cpp" nodes = [] files = [] files.append(FileNode(rev1,file1)) nodes.append(CiNode(log1, who1, date1, files)) files = [] files.append(FileNode(rev2, file2)) files.append(FileNode(rev3, file3)) nodes.append(CiNode(log2, who2, date2, files)) nodes.append(CiNode(log3, who3, date3, [])) goodParsedResult = BonsaiResult(nodes) goodUnparsedResult = """\ %s %s %s %s %s %s """ % (who1, date1, log1, rev1, file1, who2, date2, log2, rev2, file2, rev3, file3, who3, date3, log3) badUnparsedResult = deepcopy(goodUnparsedResult) badUnparsedResult = badUnparsedResult.replace("", "") invalidDateResult = deepcopy(goodUnparsedResult) invalidDateResult = invalidDateResult.replace(str(date1), "foobar") missingRevisionResult = deepcopy(goodUnparsedResult) missingRevisionResult = missingRevisionResult.replace("rev=\""+rev3+"\"", "") missingFilenameResult = deepcopy(goodUnparsedResult) missingFilenameResult = missingFilenameResult.replace(file2, "") duplicateLogResult = deepcopy(goodUnparsedResult) duplicateLogResult = re.sub(""+log1+"", "blahblah", duplicateLogResult) duplicateFilesResult = deepcopy(goodUnparsedResult) duplicateFilesResult = re.sub("\s*", "", duplicateFilesResult) missingCiResult = deepcopy(goodUnparsedResult) r = re.compile("", re.DOTALL | re.MULTILINE) missingCiResult = re.sub(r, "", missingCiResult) badResultMsgs = { 'badUnparsedResult': "BonsaiParser did not raise an exception when given a bad query", 'invalidDateResult': "BonsaiParser did not raise an exception when given an invalid date", 'missingRevisionResult': "BonsaiParser did not raise an exception when a revision was missing", 'missingFilenameResult': "BonsaiParser did not raise an exception when a filename was missing", 'duplicateLogResult': "BonsaiParser did not raise an exception when there was two tags", 'duplicateFilesResult': "BonsaiParser did not raise an exception when there was two tags", 'missingCiResult': "BonsaiParser did not raise an exception when there was no tags" } class FakeBonsaiPoller(BonsaiPoller): def __init__(self): BonsaiPoller.__init__(self, "fake url", "fake module", "fake branch") class TestBonsaiPoller(unittest.TestCase): def testFullyFormedResult(self): br = BonsaiParser(StringIO(goodUnparsedResult)) result = br.getData() # make sure the result is a BonsaiResult self.failUnless(isinstance(result, BonsaiResult)) # test for successful parsing self.failUnlessEqual(goodParsedResult, result, "BonsaiParser did not return the expected BonsaiResult") def testBadUnparsedResult(self): try: BonsaiParser(StringIO(badUnparsedResult)) self.fail(badResultMsgs["badUnparsedResult"]) except InvalidResultError: pass def testInvalidDateResult(self): try: BonsaiParser(StringIO(invalidDateResult)) self.fail(badResultMsgs["invalidDateResult"]) except InvalidResultError: pass def testMissingRevisionResult(self): try: BonsaiParser(StringIO(missingRevisionResult)) self.fail(badResultMsgs["missingRevisionResult"]) except InvalidResultError: pass def testMissingFilenameResult(self): try: BonsaiParser(StringIO(missingFilenameResult)) self.fail(badResultMsgs["missingFilenameResult"]) except InvalidResultError: pass def testDuplicateLogResult(self): try: BonsaiParser(StringIO(duplicateLogResult)) self.fail(badResultMsgs["duplicateLogResult"]) except InvalidResultError: pass def testDuplicateFilesResult(self): try: BonsaiParser(StringIO(duplicateFilesResult)) self.fail(badResultMsgs["duplicateFilesResult"]) except InvalidResultError: pass def testMissingCiResult(self): try: BonsaiParser(StringIO(missingCiResult)) self.fail(badResultMsgs["missingCiResult"]) except EmptyResult: pass def testChangeNotSubmitted(self): "Make sure a change is not submitted if the BonsaiParser fails" poller = FakeBonsaiPoller() lastChangeBefore = poller.lastChange poller._process_changes(StringIO(badUnparsedResult)) # self.lastChange will not be updated if the change was not submitted self.failUnlessEqual(lastChangeBefore, poller.lastChange) From warner at users.sourceforge.net Sun Nov 5 05:13:16 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 05 Nov 2006 05:13:16 +0000 Subject: [Buildbot-commits] buildbot/buildbot/changes bonsaipoller.py, 1.2, 1.3 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/changes In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv3175/buildbot/changes Modified Files: bonsaipoller.py Log Message: [project @ bonsaipoller: apply updates+test from Ben Hearsum. Closes SF#1590310.] Original author: warner at lothar.com Date: 2006-11-05 05:11:54 Index: bonsaipoller.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/changes/bonsaipoller.py,v retrieving revision 1.2 retrieving revision 1.3 diff -u -d -r1.2 -r1.3 --- bonsaipoller.py 30 Sep 2006 20:20:35 -0000 1.2 +++ bonsaipoller.py 5 Nov 2006 05:13:14 -0000 1.3 @@ -1,4 +1,3 @@ - import time from urllib import urlopen from xml.dom import minidom, Node @@ -9,129 +8,170 @@ from buildbot.changes import base, changes +class InvalidResultError(Exception): + def __init__(self, value="InvalidResultError"): + self.value = value + def __str__(self): + return repr(self.value) + +class EmptyResult(Exception): + pass + +class NoMoreCiNodes(Exception): + pass + +class NoMoreFileNodes(Exception): + pass class BonsaiResult: """I hold a list of CiNodes""" - def __init__(self): - self.nodes = [] + def __init__(self, nodes=[]): + self.nodes = nodes + + def __cmp__(self, other): + if len(self.nodes) != len(other.nodes): + return False + for i in range(len(self.nodes)): + if self.nodes[i].log != other.nodes[i].log \ + or self.nodes[i].who != other.nodes[i].who \ + or self.nodes[i].date != other.nodes[i].date \ + or len(self.nodes[i].files) != len(other.nodes[i].files): + return -1 + + for j in range(len(self.nodes[i].files)): + if self.nodes[i].files[j].revision \ + != other.nodes[i].files[j].revision \ + or self.nodes[i].files[j].filename \ + != other.nodes[i].files[j].filename: + return -1 + + return 0 class CiNode: - """I hold information about one Ci node, including a list of files""" - def __init__(self): - self.log = "" - self.who = "" - self.date = "" - self.files = [] + """I hold information baout one node, including a list of files""" + def __init__(self, log="", who="", date=0, files=[]): + self.log = log + self.who = who + self.date = date + self.files = files class FileNode: - """I hold information about one file node""" - def __init__(self): - self.revision = "" - self.filename = "" + """I hold information about one node""" + def __init__(self, revision="", filename=""): + self.revision = revision + self.filename = filename class BonsaiParser: - """I parse the XML result from a Bonsai cvsquery. - Typical usage is as follows:: - - bp = BonsaiParser(urlopen(bonsaiURL)) - data = bp.getData() - for cinode in data.nodes: - print cinode.who, cinode.log, cinode.date - for file in cinode.files: - print file.filename, file.revision - """ + """I parse the XML result from a bonsai cvsquery.""" def __init__(self, bonsaiQuery): try: self.dom = minidom.parse(bonsaiQuery) except: - self.dom = None - return - self.currentCiNode = None - self.currentFileNode = None + raise InvalidResultError("Malformed XML in result") + + self.ciNodes = self.dom.getElementsByTagName("ci") + self.currentCiNode = None # filled in by _nextCiNode() + self.fileNodes = None # filled in by _nextCiNode() + self.currentFileNode = None # filled in by _nextFileNode() + self.bonsaiResult = self._parseData() def getData(self): - """I return data from a Bonsai cvsquery""" - data = BonsaiResult() - while self._nextCiNode(): - ci = CiNode() - ci.log = self._getLog() - ci.who = self._getWho() - ci.date = self._getDate() - while self._nextFileNode(): - fn = FileNode() - fn.revision = self._getRevision() - fn.filename = self._getFilename() - ci.files.append(fn) + return self.bonsaiResult - data.nodes.append(ci) + def _parseData(self): + """Returns data from a Bonsai cvsquery in a BonsaiResult object""" + nodes = [] + try: + while self._nextCiNode(): + files = [] + try: + while self._nextFileNode(): + files.append(FileNode(self._getRevision(), + self._getFilename())) + except NoMoreFileNodes: + pass + except InvalidResultError: + raise + nodes.append(CiNode(self._getLog(), self._getWho(), + self._getDate(), files)) - return data + except NoMoreCiNodes: + pass + except InvalidResultError, EmptyResult: + raise + + return BonsaiResult(nodes) def _nextCiNode(self): + """Iterates to the next node and fills self.fileNodes with + child nodes""" try: - # first node? + self.currentCiNode = self.ciNodes.pop(0) + if len(self.currentCiNode.getElementsByTagName("files")) > 1: + raise InvalidResultError("Multiple for one ") + + self.fileNodes = self.currentCiNode.getElementsByTagName("f") + except IndexError: + # if there was zero nodes in the result if not self.currentCiNode: - # every other sibling is a , so jump 2 ahead - self.currentCiNode = self.dom.getElementsByTagName("ci")[0] + raise EmptyResult else: - self.currentCiNode = self.currentCiNode.nextSibling.nextSibling - except (AttributeError,IndexError): - self.currentCiNode = None + raise NoMoreCiNodes - if self.currentCiNode: - return True - else: - return False + return True + + def _nextFileNode(self): + """Iterates to the next node""" + try: + self.currentFileNode = self.fileNodes.pop(0) + except IndexError: + raise NoMoreFileNodes + + return True def _getLog(self): - log = "" - for child in self.currentCiNode.childNodes: - if child.nodeType == Node.ELEMENT_NODE and child.tagName == "log": - log = child.firstChild.data - return str(log) + """Returns the log of the current node""" + logs = self.currentCiNode.getElementsByTagName("log") + if len(logs) < 1: + raise InvalidResultError("No log present") + elif len(logs) > 1: + raise InvalidResultError("Multiple logs present") + return logs[0].firstChild.data def _getWho(self): - """Returns the e-mail address of the commit'er""" - return str(self.currentCiNode.getAttribute("who").replace("%", "@")) + """Returns the e-mail address of the commiter""" + # convert unicode string to regular string + return str(self.currentCiNode.getAttribute("who")) def _getDate(self): """Returns the date (unix time) of the commit""" - return int(self.currentCiNode.getAttribute("date")) - + # convert unicode number to regular one + try: + commitDate = int(self.currentCiNode.getAttribute("date")) + except ValueError: + raise InvalidResultError - def _firstFileNode(self): - for child in self.currentCiNode.childNodes: - if child.nodeType == Node.ELEMENT_NODE and child.tagName == "files": - # child is now the element - for c in child.childNodes: - if c.nodeType == Node.ELEMENT_NODE and c.tagName == "f": - return c + return commitDate - def _nextFileNode(self): - # every other sibling is a , so go two past the current one + def _getFilename(self): + """Returns the filename of the current node""" try: - # first node? - if not self.currentFileNode: - self.currentFileNode = self._firstFileNode() - else: - self.currentFileNode = self.currentFileNode.nextSibling.nextSibling + filename = self.currentFileNode.firstChild.data except AttributeError: - self.currentFileNode = None - - if self.currentFileNode: - return True - else: - return False + raise InvalidResultError("Missing filename") - def _getFilename(self): - return str(self.currentFileNode.firstChild.data) + return filename def _getRevision(self): - return str(self.currentFileNode.getAttribute("rev")) + """Returns the revision of the current node""" + rev = self.currentFileNode.getAttribute("rev") + if rev == "": + raise InvalidResultError("A revision was missing from a file") + return rev class BonsaiPoller(base.ChangeSource): @@ -215,7 +255,7 @@ log.msg("Bonsai poll failed: %s" % res) return res - def _get_changes(self): + def _make_url(self): args = ["treeid=%s" % self.tree, "module=%s" % self.module, "branch=%s" % self.branch, "branchtype=match", "sortby=Date", "date=explicit", @@ -226,6 +266,11 @@ url = self.bonsaiURL url += "/cvsquery.cgi?" url += "&".join(args) + + return url + + def _get_changes(self): + url = self._make_url() log.msg("Polling Bonsai tree at %s" % url) self.lastPoll = time.time() @@ -233,10 +278,17 @@ return defer.maybeDeferred(urlopen, url) def _process_changes(self, query): - bp = BonsaiParser(query) files = [] - data = bp.getData() - for cinode in data.nodes: + try: + bp = BonsaiParser(query) + result = bp.getData() + except InvalidResultError, e: + log.msg("Could not process Bonsai query: " + e.value) + return + except EmptyResult: + return + + for cinode in result.nodes: for file in cinode.files: files.append(file.filename+' (revision '+file.revision+')') c = changes.Change(who = cinode.who, @@ -246,4 +298,3 @@ branch = self.branch) self.parent.addChange(c) self.lastChange = self.lastPoll - From warner at users.sourceforge.net Sun Nov 19 06:46:36 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 19 Nov 2006 06:46:36 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.772,1.773 NEWS,1.58,1.59 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv13418 Modified Files: ChangeLog NEWS Log Message: [project @ NEWS: start collecting items for the next release] Original author: warner at lothar.com Date: 2006-11-19 06:45:14 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.772 retrieving revision 1.773 diff -u -d -r1.772 -r1.773 --- ChangeLog 5 Nov 2006 05:13:14 -0000 1.772 +++ ChangeLog 19 Nov 2006 06:46:33 -0000 1.773 @@ -1,3 +1,7 @@ +2006-11-18 Brian Warner + + * NEWS: start collecting items for the next release. + 2006-11-04 Brian Warner * buildbot/changes/bonsaipoller.py: apply updates from Ben Index: NEWS =================================================================== RCS file: /cvsroot/buildbot/buildbot/NEWS,v retrieving revision 1.58 retrieving revision 1.59 diff -u -d -r1.58 -r1.59 --- NEWS 23 Aug 2006 07:13:36 -0000 1.58 +++ NEWS 19 Nov 2006 06:46:33 -0000 1.59 @@ -1,5 +1,96 @@ User visible changes in Buildbot. +* Release ?.?.? (?) + +** Things You Need To Know + +*** The Great BuildStep Renaming + +All BuildSteps have moved from being classes in buildbot.process.step to +separate modules in buildbot.steps.* . They have been split out into separate +categories: for example, the source checkout steps are now +buildbot.steps.source.CVS, buildbot.steps.source.Darcs, etc. The most +commonly used one is probably buildbot.steps.shell.ShellCommand . The +python-specific steps are in buildbot.steps.python, and the Twisted-specific +steps are in buildbot.steps.python_twisted . + +The old names are deprecated and will be removed altogether in the next +release. + +** new features + +*** Locks now take maxCount=N to allow multiple simultaneous owners + +This allows Locks to be non-exclusive but still limit concurrency. Thanks to +James Knight for the patch. Closes SF#1434997. + +*** filetransfer steps + +buildbot.steps.transfer.FileUpload is a buildstep that will move files from +the slave to the master. Likewise, FileDownload will move files from the +master down to the buildslave. Many thanks to Albert Hofkamp for contributing +these classes. Closes SF#1504631. + +*** pyflakes step + +buildbot.steps.python.PyFlakes will run the simple 'pyflakes' static analysis +tool and parse the results to tell you about undefined names, unused imports, +etc. + +*** Monotone support + +Nathaniel Smith has contributed initial support for the Monotone version +control system. The code still needs docs and tests, but on the other hand it +has been in use by the Monotone buildbot for a long time now, so it is +probably fairly stable. + +*** Tinderbox support + +Ben Hearsum and the Mozilla crew have contributed some classes to allow +Buildbot to work with Tinderbox clients. One piece is +buildbot.changes.bonsaipoller.BonsaiPoller, which is a ChangeSource that +polls a Bonsai server (which is a kind of web-vased viewcvs CGI script) to +discover source code changes. The other piece is +buildbot.status.tinderbox.TinderboxMailNotifier, which is a status plugin +that sends email in the same format as Tinderbox does, which allows a number +of Tinderbox tools to be driven by Buildbot instead. + +*** SVN Poller + +Niklaus Giger contributed a ChangeSource (buildbot.changes.svnpoller) which +polls a remote SVN repository on a periodic basis. This is useful when, for +whatever reason, you cannot add a post-commit hook script to the repository. +This obsoletes the external contrib/svn_watcher.py script. + +** bug fixes + +*** Update source.SVN to work with the new SVN-1.4.0 + +The latest subversion changed the behavior in an unusual situation which +caused the unit tests to fail. This was unlikely to cause a problem in actual +usage, but the tests have been updated to pass with the new version. + +*** update svn_buildbot.py to avoid mangling filenames + +Older versions of this script were stripping the wrong number of columns from +the output of 'svnlook changed', and would sometimes mangle filenames. This +has been fixed. Closes SF#1545146. + +*** logfiles= caused subsequent build failures under Windows + +Earlier versions of buildbot didn't explicitly close any logfiles= file +handles when the build finished. On windows (where you cannot delete a file +that someone is reading), this could cause the next build to fail as the +source checkout step was unable to delete the old working directory. This has +been fixed. Closes SF#1568415. + +*** logfiles= didn't work on OS-X + +Macintosh OS-X has a different behavior when reading files that have reached +EOF, the result was that logfiles= sometimes didn't work. Thanks to Mark Rowe +for the patch. + + * Release 0.7.4 (23 Aug 2006) ** Things You Need To Know From warner at users.sourceforge.net Sun Nov 19 20:22:22 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 19 Nov 2006 20:22:22 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.773,1.774 NEWS,1.59,1.60 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv8523 Modified Files: ChangeLog NEWS Log Message: [project @ more NEWS items] Original author: warner at lothar.com Date: 2006-11-19 20:21:47 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.773 retrieving revision 1.774 diff -u -d -r1.773 -r1.774 --- ChangeLog 19 Nov 2006 06:46:33 -0000 1.773 +++ ChangeLog 19 Nov 2006 20:22:20 -0000 1.774 @@ -1,3 +1,7 @@ +2006-11-19 Brian Warner + + * NEWS (IStatusLog.readlines): more news items + 2006-11-18 Brian Warner * NEWS: start collecting items for the next release. Index: NEWS =================================================================== RCS file: /cvsroot/buildbot/buildbot/NEWS,v retrieving revision 1.59 retrieving revision 1.60 diff -u -d -r1.59 -r1.60 --- NEWS 19 Nov 2006 06:46:33 -0000 1.59 +++ NEWS 19 Nov 2006 20:22:20 -0000 1.60 @@ -62,6 +62,28 @@ whatever reason, you cannot add a post-commit hook script to the repository. This obsoletes the external contrib/svn_watcher.py script. +** notes for plugin developers + +*** IStatusLog.readlines() + +This new method makes it easier for a status plugin (or a +BuildStep.createSummary method) to walk through a StatusLog one line at a +time. For example, if you wanted to create an extra logfile that just +contained all the GCC warnings from the main log, you could use the +following: + + def createSummary(self, log): + warnings = [] + for line in log.readlines(): + if "warning:" in line: + warnings.append() + self.addCompleteLog('warnings', "".join(warnings)) + +The "BuildStep LogFiles" section of the user's manual contains more +information. This method is not particularly memory-efficient yet (it reads +the whole logfile into memory first, then splits it into lines); this will be +improved in a future release. + ** bug fixes *** Update source.SVN to work with the new SVN-1.4.0 From warner at users.sourceforge.net Fri Nov 24 07:13:55 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:13:55 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.774,1.775 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv31432 Modified Files: ChangeLog Log Message: [project @ svn_buildbot.py: don't use /usr/bin/env, to allow use of python2.3 *or* 2.4] Original author: warner at lothar.com Date: 2006-11-23 20:24:38 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.774 retrieving revision 1.775 diff -u -d -r1.774 -r1.775 --- ChangeLog 19 Nov 2006 20:22:20 -0000 1.774 +++ ChangeLog 24 Nov 2006 07:13:53 -0000 1.775 @@ -1,3 +1,9 @@ +2006-11-23 Brian Warner + + * contrib/svn_buildbot.py: use /usr/bin/python, not /usr/bin/env, + to allow use of python2.4 or whatever. This tool still requires + python2.3 or newer. + 2006-11-19 Brian Warner * NEWS (IStatusLog.readlines): more news items From warner at users.sourceforge.net Fri Nov 24 07:13:56 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:13:56 +0000 Subject: [Buildbot-commits] buildbot/contrib svn_buildbot.py,1.13,1.14 Message-ID: Update of /cvsroot/buildbot/buildbot/contrib In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv31432/contrib Modified Files: svn_buildbot.py Log Message: [project @ svn_buildbot.py: don't use /usr/bin/env, to allow use of python2.3 *or* 2.4] Original author: warner at lothar.com Date: 2006-11-23 20:24:38 Index: svn_buildbot.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/contrib/svn_buildbot.py,v retrieving revision 1.13 retrieving revision 1.14 diff -u -d -r1.13 -r1.14 --- svn_buildbot.py 25 Sep 2006 08:01:23 -0000 1.13 +++ svn_buildbot.py 24 Nov 2006 07:13:53 -0000 1.14 @@ -1,4 +1,4 @@ -#!/usr/bin/env python2.3 +#!/usr/bin/python # this requires python >=2.3 for the 'sets' module. From warner at users.sourceforge.net Fri Nov 24 07:16:37 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:16:37 +0000 Subject: [Buildbot-commits] buildbot/buildbot master.py,1.97,1.98 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv32609/buildbot Modified Files: master.py Log Message: [project @ reconfig no longer interrupts builds, nor does it disconnect/reconnect slaves] Original author: warner at lothar.com Date: 2006-11-23 21:32:36 Index: master.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/master.py,v retrieving revision 1.97 retrieving revision 1.98 diff -u -d -r1.97 -r1.98 --- master.py 17 Sep 2006 20:43:39 -0000 1.97 +++ master.py 24 Nov 2006 07:16:35 -0000 1.98 @@ -1,6 +1,5 @@ # -*- test-case-name: buildbot.test.test_run -*- -from __future__ import generators import string, os signal = None try: @@ -41,36 +40,19 @@ reference to this instance. The BotMaster object is stashed as the .service attribute.""" - slave_commands = None - - def __init__(self, name): + def __init__(self, name, botmaster): self.slavename = name + self.botmaster = botmaster self.slave_status = SlaveStatus(name) - self.builders = [] # list of b.p.builder.Builder instances self.slave = None # a RemoteReference to the Bot, when connected + self.slave_commands = None - def addBuilder(self, builder): - """Called to add a builder after the slave has connected. + def updateSlave(self): + """Called to add or remove builders after the slave has connected. @return: a Deferred that indicates when an attached slave has - accepted the new builder.""" - - self.builders.append(builder) - if self.slave: - return self.sendBuilderList() - return defer.succeed(None) - - def removeBuilder(self, builder): - """Tell the slave that the given builder has been removed, allowing - it to discard the associated L{buildbot.slave.bot.SlaveBuilder} - object. - - @return: a Deferred that fires when the slave has finished removing - the SlaveBuilder - """ - self.builders.remove(builder) + accepted the new builders and/or released the old ones.""" if self.slave: - builder.detached(self) return self.sendBuilderList() return defer.succeed(None) @@ -79,7 +61,7 @@ (self.slavename, string.join(map(lambda b: b.name, self.builders), ',')) - def attached(self, mind): + def attached(self, bot): """This is called when the slave connects. @return: a Deferred that fires with a suitable pb.IPerspective to @@ -98,14 +80,91 @@ # squabble tport = self.slave.broker.transport log.msg("old slave was connected from", tport.getPeer()) - log.msg("new slave is from", mind.broker.transport.getPeer()) + log.msg("new slave is from", bot.broker.transport.getPeer()) d = self.disconnect() - d.addCallback(lambda res: self._attached(mind)) - return d + else: + d = defer.succeed(None) + # now we go through a sequence of calls, gathering information, then + # tell the Botmaster that it can finally give this slave to all the + # Builders that care about it. + + # we accumulate slave information in this 'state' dictionary, then + # set it atomically if we make it far enough through the process + state = {} + + def _log_attachment_on_slave(res): + d1 = bot.callRemote("print", "attached") + d1.addErrback(lambda why: None) + return d1 + d.addCallback(_log_attachment_on_slave) + + def _get_info(res): + d1 = bot.callRemote("getSlaveInfo") + def _got_info(info): + log.msg("Got slaveinfo from '%s'" % self.slavename) + # TODO: info{} might have other keys + state["admin"] = info.get("admin") + state["host"] = info.get("host") + def _info_unavailable(why): + # maybe an old slave, doesn't implement remote_getSlaveInfo + log.msg("BotPerspective.info_unavailable") + log.err(why) + d1.addCallbacks(_got_info, _info_unavailable) + return d1 + d.addCallback(_get_info) + + def _get_commands(res): + d1 = bot.callRemote("getCommands") + def _got_commands(commands): + state["slave_commands"] = commands + def _commands_unavailable(why): + # probably an old slave + log.msg("BotPerspective._commands_unavailable") + if why.check(AttributeError): + return + log.err(why) + d1.addCallbacks(_got_commands, _commands_unavailable) + return d1 + d.addCallback(_get_commands) + + def _accept_slave(res): + self.slave_status.setAdmin(state.get("admin")) + self.slave_status.setHost(state.get("host")) + self.slave_status.setConnected(True) + self.slave_commands = state.get("slave_commands") + self.slave = bot + log.msg("bot attached") + return self.updateSlave() + d.addCallback(_accept_slave) + + # Finally, the slave gets a reference to this BotPerspective. They + # receive this later, after we've started using them. + d.addCallback(lambda res: self) + return d + + def detached(self, mind): + self.slave = None + self.slave_status.setConnected(False) + self.botmaster.slaveLost(self) + log.msg("BotPerspective.detached(%s)" % self.slavename) - return self._attached(mind) def disconnect(self): + """Forcibly disconnect the slave. + + This severs the TCP connection and returns a Deferred that will fire + (with None) when the connection is probably gone. + + If the slave is still alive, they will probably try to reconnect + again in a moment. + + This is called in two circumstances. The first is when a slave is + removed from the config file. In this case, when they try to + reconnect, they will be rejected as an unknown slave. The second is + when we wind up with two connections for the same slave, in which + case we disconnect the older connection. + """ + if not self.slave: return defer.succeed(None) log.msg("disconnecting old slave %s now" % self.slavename) @@ -118,8 +177,11 @@ # notifyOnDisconnect handlers have had a chance to run. d = defer.Deferred() - self.slave.notifyOnDisconnect(lambda res: # TODO: d=d ? - reactor.callLater(0, d.callback, None)) + # notifyOnDisconnect runs the callback with one argument, the + # RemoteReference being disconnected. + def _disconnected(rref): + reactor.callLater(0, d.callback, None) + self.slave.notifyOnDisconnect(_disconnected) tport = self.slave.broker.transport # this is the polite way to request that a socket be closed tport.loseConnection() @@ -144,109 +206,27 @@ # When this Deferred fires, we'll be ready to accept the new slave return d - def _attached(self, mind): - """We go through a sequence of calls, gathering information, then - tell our Builders that they have a slave to work with. - - @return: a Deferred that fires (with 'self') when our Builders are - prepared to deal with the slave. - """ - self.slave = mind - d = self.slave.callRemote("print", "attached") - d.addErrback(lambda why: 0) - self.slave_status.connected = True - log.msg("bot attached") - - # TODO: there is a window here (while we're retrieving slaveinfo) - # during which a disconnect or a duplicate-slave will be confusing - d.addCallback(lambda res: self.slave.callRemote("getSlaveInfo")) - d.addCallbacks(self.got_info, self.infoUnavailable) - d.addCallback(self._attached2) - d.addCallback(lambda res: self) - return d - - def got_info(self, info): - log.msg("Got slaveinfo from '%s'" % self.slavename) - # TODO: info{} might have other keys - self.slave_status.admin = info.get("admin") - self.slave_status.host = info.get("host") - - def infoUnavailable(self, why): - # maybe an old slave, doesn't implement remote_getSlaveInfo - log.msg("BotPerspective.infoUnavailable") - log.err(why) - - def _attached2(self, res): - d = self.slave.callRemote("getCommands") - d.addCallback(self.got_commands) - d.addErrback(self._commandsUnavailable) - d.addCallback(self._attached3) - return d - - def got_commands(self, commands): - self.slave_commands = commands - - def _commandsUnavailable(self, why): - # probably an old slave - log.msg("BotPerspective._commandsUnavailable") - if why.check(AttributeError): - return - log.err(why) - - def _attached3(self, res): - d = self.slave.callRemote("getDirs") - d.addCallback(self.got_dirs) - d.addErrback(self._dirsFailed) - d.addCallback(self._attached4) - return d - - def got_dirs(self, dirs): - wanted = map(lambda b: b.builddir, self.builders) - unwanted = [] - for d in dirs: - if d not in wanted and d != "info": - unwanted.append(d) - if unwanted: - log.msg("slave %s has leftover directories (%s): " % \ - (self.slavename, string.join(unwanted, ',')) + \ - "you can delete them now") - - def _dirsFailed(self, why): - log.msg("BotPerspective._dirsFailed") - log.err(why) - - def _attached4(self, res): - return self.sendBuilderList() - def sendBuilderList(self): - # now make sure their list of Builders matches ours - blist = [] - for b in self.builders: - blist.append((b.name, b.builddir)) + our_builders = self.botmaster.getBuildersForSlave(self.slavename) + blist = [(b.name, b.builddir) for b in our_builders] d = self.slave.callRemote("setBuilderList", blist) - d.addCallback(self.list_done) - d.addErrback(self._listFailed) + def _sent(slist): + dl = [] + for name, remote in slist.items(): + # use get() since we might have changed our mind since then + b = self.botmaster.builders.get(name) + if b: + d1 = b.attached(self, remote, self.slave_commands) + dl.append(d1) + return defer.DeferredList(dl) + def _set_failed(why): + log.msg("BotPerspective.sendBuilderList (%s) failed" % self) + log.err(why) + # TODO: hang up on them?, without setBuilderList we can't use + # them + d.addCallbacks(_sent, _set_failed) return d - def list_done(self, blist): - # this could come back at weird times. be prepared to handle oddness - dl = [] - for name, remote in blist.items(): - for b in self.builders: - if b.name == name: - # if we sent the builders list because of a config - # change, the Builder might already be attached. - # Builder.attached will ignore us if this happens. - d = b.attached(self, remote, self.slave_commands) - dl.append(d) - continue - return defer.DeferredList(dl) - - def _listFailed(self, why): - log.msg("BotPerspective._listFailed") - log.err(why) - # TODO: hang up on them, without setBuilderList we can't use them - def perspective_forceBuild(self, name, who=None): # slave admins are allowed to force any of their own builds for b in self.builders: @@ -265,13 +245,6 @@ def perspective_keepalive(self): pass - def detached(self, mind): - self.slave = None - self.slave_status.connected = False - for b in self.builders: - b.detached(self) - log.msg("Botmaster.detached(%s)" % self.slavename) - class BotMaster(service.Service): @@ -341,7 +314,7 @@ def addSlave(self, slavename): - slave = BotPerspective(slavename) + slave = BotPerspective(slavename, self) self.slaves[slavename] = slave def removeSlave(self, slavename): @@ -349,54 +322,44 @@ del self.slaves[slavename] return d - def getBuildernames(self): - return self.builderNames - - def addBuilder(self, builder): - """This is called by the setup code to define what builds should be - performed. Each Builder object has a build slave that should host - that build: the builds cannot be done until the right slave - connects. + def slaveLost(self, bot): + for name, b in self.builders.items(): + if bot.slavename in b.slavenames: + b.detached(bot) - @return: a Deferred that fires when an attached slave has accepted - the new builder. - """ + def getBuildersForSlave(self, slavename): + return [b + for b in self.builders.values() + if slavename in b.slavenames] - if self.debug: print "addBuilder", builder - log.msg("Botmaster.addBuilder(%s)" % builder.name) + def getBuildernames(self): + return self.builderNames - if builder.name in self.builderNames: - raise KeyError("muliply defined builder '%s'" % builder.name) - for slavename in builder.slavenames: - if not self.slaves.has_key(slavename): - raise KeyError("builder %s uses undefined slave %s" % \ - (builder.name, slavename)) + def getBuilders(self): + allBuilders = [self.builders[name] for name in self.builderNames] + return allBuilders - self.builders[builder.name] = builder - self.builderNames.append(builder.name) - builder.setBotmaster(self) + def setBuilders(self, builders): + self.builders = {} + self.builderNames = [] + for b in builders: + for slavename in b.slavenames: + # this is actually validated earlier + assert slavename in self.slaves + self.builders[b.name] = b + self.builderNames.append(b.name) + b.setBotmaster(self) + d = self._updateAllSlaves() + return d - dl = [self.slaves[slavename].addBuilder(builder) - for slavename in builder.slavenames] + def _updateAllSlaves(self): + """Notify all buildslaves about changes in their Builders.""" + dl = [s.updateSlave() for s in self.slaves.values()] return defer.DeferredList(dl) - def removeBuilder(self, builder): - """Stop using a Builder. - This removes the Builder from the list of active Builders. - - @return: a Deferred that fires when an attached slave has finished - removing the SlaveBuilder - """ - if self.debug: print "removeBuilder", builder - log.msg("Botmaster.removeBuilder(%s)" % builder.name) - b = self.builders[builder.name] - del self.builders[builder.name] - self.builderNames.remove(builder.name) - for slavename in builder.slavenames: - slave = self.slaves.get(slavename) - if slave: - return slave.removeBuilder(builder) - return defer.succeed(None) + def maybeStartAllBuilds(self): + for b in self.builders.values(): + b.maybeStartBuild() def getPerspective(self, slavename): return self.slaves[slavename] @@ -874,9 +837,11 @@ self.slavePortnum = slavePortnum log.msg("configuration update started") - d.addCallback(lambda res: log.msg("configuration update complete")) - self.readConfig = True # TODO: consider not setting this until the - # Deferred fires. + def _done(res): + self.readConfig = True + log.msg("configuration update complete") + d.addCallback(_done) + d.addCallback(lambda res: self.botmaster.maybeStartAllBuilds()) return d def loadConfig_Slaves(self, bots): @@ -932,29 +897,28 @@ d.addCallback(addNewOnes) return d - def loadConfig_Builders(self, newBuilders): - dl = [] - old = self.botmaster.getBuildernames() - newNames = [] + def loadConfig_Builders(self, newBuilderData): + somethingChanged = False newList = {} - for data in newBuilders: + newBuilderNames = [] + allBuilders = self.botmaster.builders.copy() + for data in newBuilderData: name = data['name'] newList[name] = data - newNames.append(name) + newBuilderNames.append(name) # identify all that were removed - for old in self.botmaster.builders.values()[:]: - if old.name not in newList.keys(): - log.msg("removing old builder %s" % old.name) - d = self.botmaster.removeBuilder(old) - dl.append(d) + for oldname in self.botmaster.getBuildernames(): + if oldname not in newList: + log.msg("removing old builder %s" % oldname) + del allBuilders[oldname] + somethingChanged = True # announce the change - self.status.builderRemoved(old.name) + self.status.builderRemoved(oldname) # everything in newList is either unchanged, changed, or new - for newName, data in newList.items(): - old = self.botmaster.builders.get(newName) - name = data['name'] + for name, data in newList.items(): + old = self.botmaster.builders.get(name) basedir = data['builddir'] # used on both master and slave #name, slave, builddir, factory = data if not old: # new @@ -964,35 +928,38 @@ (name, category)) statusbag = self.status.builderAdded(name, basedir, category) builder = Builder(data, statusbag) - d = self.botmaster.addBuilder(builder) - dl.append(d) - else: + allBuilders[name] = builder + somethingChanged = True + elif old.compareToSetup(data): + # changed: try to minimize the disruption and only modify the + # pieces that really changed diffs = old.compareToSetup(data) - if not diffs: # unchanged: leave it alone - log.msg("builder %s is unchanged" % name) - pass - else: - # changed: remove and re-add. Don't touch the statusbag - # object: the clients won't see a remove/add cycle - log.msg("updating builder %s: %s" % (name, - "\n".join(diffs))) - # TODO: if the basedir was changed, we probably need to - # make a new statusbag - # TODO: if a slave is connected and we're re-using the - # same slave, try to avoid a disconnect/reconnect cycle. - statusbag = old.builder_status - statusbag.saveYourself() # seems like a good idea - d = self.botmaster.removeBuilder(old) - dl.append(d) - builder = Builder(data, statusbag) - # point out that the builder was updated - statusbag.addPointEvent(["config", "updated"]) - d = self.botmaster.addBuilder(builder) - dl.append(d) - # now that everything is up-to-date, make sure the names are in the - # desired order - self.botmaster.builderNames = newNames - return defer.DeferredList(dl, fireOnOneErrback=1, consumeErrors=0) + log.msg("updating builder %s: %s" % (name, "\n".join(diffs))) + + statusbag = old.builder_status + statusbag.saveYourself() # seems like a good idea + # TODO: if the basedir was changed, we probably need to make + # a new statusbag + new_builder = Builder(data, statusbag) + new_builder.consumeTheSoulOfYourPredecessor(old) + # that migrates any retained slavebuilders too + + # point out that the builder was updated. On the Waterfall, + # this will appear just after any currently-running builds. + statusbag.addPointEvent(["config", "updated"]) + + allBuilders[name] = new_builder + somethingChanged = True + else: + # unchanged: leave it alone + log.msg("builder %s is unchanged" % name) + pass + + if somethingChanged: + sortedAllBuilders = [allBuilders[name] for name in newBuilderNames] + d = self.botmaster.setBuilders(sortedAllBuilders) + return d + return None def loadConfig_status(self, status): dl = [] From warner at users.sourceforge.net Fri Nov 24 07:16:38 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:16:38 +0000 Subject: [Buildbot-commits] buildbot/docs buildbot.texinfo,1.84,1.85 Message-ID: Update of /cvsroot/buildbot/buildbot/docs In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv32609/docs Modified Files: buildbot.texinfo Log Message: [project @ reconfig no longer interrupts builds, nor does it disconnect/reconnect slaves] Original author: warner at lothar.com Date: 2006-11-23 21:32:36 Index: buildbot.texinfo =================================================================== RCS file: /cvsroot/buildbot/buildbot/docs/buildbot.texinfo,v retrieving revision 1.84 retrieving revision 1.85 diff -u -d -r1.84 -r1.85 --- buildbot.texinfo 13 Oct 2006 09:14:07 -0000 1.84 +++ buildbot.texinfo 24 Nov 2006 07:16:36 -0000 1.85 @@ -1903,6 +1903,16 @@ password-protected button on the web page, as well as a privileged IRC command). +When reloading the config file, the buildmaster will endeavor to +change as little as possible about the running system. For example, +although old status targets may be shut down and new ones started up, +any status targets that were not changed since the last time the +config file was read will be left running and untouched. Likewise any +Builders which have not been changed will be left running. If a +Builder is modified (say, the build process is changed) while a Build +is currently running, that Build will keep running with the old +process until it completes. Any previously queued Builds (or Builds +which get queued after the reconfig) will use the new process. @node Defining the Project, Listing Change Sources and Schedulers, Loading the Config File, Configuration @section Defining the Project From warner at users.sourceforge.net Fri Nov 24 07:16:37 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:16:37 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.775,1.776 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv32609 Modified Files: ChangeLog Log Message: [project @ reconfig no longer interrupts builds, nor does it disconnect/reconnect slaves] Original author: warner at lothar.com Date: 2006-11-23 21:32:36 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.775 retrieving revision 1.776 diff -u -d -r1.775 -r1.776 --- ChangeLog 24 Nov 2006 07:13:53 -0000 1.775 +++ ChangeLog 24 Nov 2006 07:16:35 -0000 1.776 @@ -1,5 +1,67 @@ 2006-11-23 Brian Warner + * buildbot/master.py (BuildMaster.loadConfig_Builders): changing a + Builder no longer induces a disconnect/reconnect cycle. This means + that any builds currently in progress will not be interrupted, and + any builds which are queued in the Builder will not be lost. This + is implemented by having the new Builder extract the state (i.e. + all pending Builds and any desired SlaveBuilders) from the old + Builder. + (BotPerspective): refactor. The BotPerspective no longer keeps + track of all the Builders that want to use this slave; instead, it + asks the BotMaster each time it needs this list. This removes + addBuilder and removeBuilder. Clean up attached() to acquire all + the slave's information in a more atomic fashion. updateSlave() is + now the way to make sure the slave is using the right set of + Builders: just call it after everything else has been + reconfigured. + (BotMaster): refactor, removing addBuilder/removeBuilder and + replacing them with an all-at-once setBuilders() call. + + * buildbot/test/test_slaves.py (Reconfig): new test case to + exercise this functionality + * buildbot/steps/dummy.py (Wait): new dummy BuildStep for the test + * buildbot/slave/commands.py (WaitCommand): same + + * docs/buildbot.texinfo (Loading the Config File): document the + changes + + * buildbot/process/builder.py (SlaveBuilder): refactor. Allow the + SlaveBuilder to have its parent Builder changed. + (SlaveBuilder.isAvailable): new method to give access to state, + which is now a private attribute + (SlaveBuilder.buildStarted,buildFinished): new methods to inform + the SlaveBuilder about how it is being used. These methods update + its internal state. buildFinished() is now the place that invokes + maybeStartBuild() on its parent Builder. + (Builder.consumeTheSoulOfYourPredecessor): new method to allow a + new Builder to take over for an old one, transferring state from + the old one. + (Buider): refactor the way that SlaveBuilders are used to match, + giving them a bit more autonomy. + (Builder.buildFinished): this no longer calls maybeStartBuild(): + instead the SlaveBuilder calls it on whoever its parent Builder is + at the time. This way, when an old Builder is replaced by a new + one, and there was a build in progress during the transition, when + that build finishes, it will be the new Builder that is told about + the newly available slave so it can start a new build. + + * buildbot/process/base.py (Build.startBuild._release_slave): when + the Build finishes, tell the SlaveBuilder that they've been + released. + + * buildbot/status/builder.py (SlaveStatus): add some new setter + methods for use by BotPerspective, to keep some attributes more + private + + * buildbot/slave/bot.py (Bot.remote_getDirs): this is no longer + called by the buildmaster. + (Bot.setBuilderList): instead, we locally announce any leftover + directories based upon which Builders we were told about. The + master doesn't really care; it's the local admin who may or may not + wish to delete them. + + * contrib/svn_buildbot.py: use /usr/bin/python, not /usr/bin/env, to allow use of python2.4 or whatever. This tool still requires python2.3 or newer. From warner at users.sourceforge.net Fri Nov 24 07:16:37 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:16:37 +0000 Subject: [Buildbot-commits] buildbot/buildbot/process base.py, 1.71, 1.72 builder.py, 1.37, 1.38 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/process In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv32609/buildbot/process Modified Files: base.py builder.py Log Message: [project @ reconfig no longer interrupts builds, nor does it disconnect/reconnect slaves] Original author: warner at lothar.com Date: 2006-11-23 21:32:36 Index: base.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/process/base.py,v retrieving revision 1.71 retrieving revision 1.72 diff -u -d -r1.71 -r1.72 --- base.py 25 Sep 2006 02:43:56 -0000 1.71 +++ base.py 24 Nov 2006 07:16:35 -0000 1.72 @@ -299,6 +299,10 @@ self.remote = slavebuilder.remote self.remote.notifyOnDisconnect(self.lostRemote) d = self.deferred = defer.Deferred() + def _release_slave(res): + self.slavebuilder.buildFinished() + return res + d.addCallback(_release_slave) try: self.setupBuild(expectations) # create .steps Index: builder.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/process/builder.py,v retrieving revision 1.37 retrieving revision 1.38 diff -u -d -r1.37 -r1.38 --- builder.py 6 Sep 2006 00:41:55 -0000 1.37 +++ builder.py 24 Nov 2006 07:16:35 -0000 1.38 @@ -24,13 +24,14 @@ buildbot. When a remote builder connects, I query it for command versions and then make it available to any Builds that are ready to run. """ - state = ATTACHING - remote = None - build = None - - def __init__(self, builder): - self.builder = builder + def __init__(self): self.ping_watchers = [] + self.state = ATTACHING + self.remote = None + + def setBuilder(self, b): + self.builder = b + self.builder_name = b.name def getSlaveCommandVersion(self, command, oldversion=None): if self.remoteCommands is None: @@ -38,12 +39,17 @@ return oldversion return self.remoteCommands.get(command) + def isAvailable(self): + if self.state == IDLE: + return True + return False + def attached(self, slave, remote, commands): self.slave = slave self.remote = remote self.remoteCommands = commands # maps command name to version log.msg("Buildslave %s attached to %s" % (slave.slavename, - self.builder.name)) + self.builder_name)) d = self.remote.callRemote("setMaster", self) d.addErrback(self._attachFailure, "Builder.setMaster") d.addCallback(self._attached2) @@ -57,6 +63,7 @@ def _attached3(self, res): # now we say they're really attached + self.state = IDLE return self def _attachFailure(self, why, where): @@ -67,17 +74,17 @@ def detached(self): log.msg("Buildslave %s detached from %s" % (self.slave.slavename, - self.builder.name)) + self.builder_name)) self.slave = None self.remote = None self.remoteCommands = None - def startBuild(self, build): - self.build = build - - def finishBuild(self): - self.build = None + def buildStarted(self): + self.state = BUILDING + def buildFinished(self): + self.state = IDLE + reactor.callLater(0, self.builder.maybeStartBuild) def ping(self, timeout, status=None): """Ping the slave to make sure it is still there. Returns a Deferred @@ -87,6 +94,7 @@ event will be pushed. """ + self.state = PINGING newping = not self.ping_watchers d = defer.Deferred() self.ping_watchers.append(d) @@ -287,7 +295,7 @@ return diffs def __repr__(self): - return "" % self.name + return "" % (self.name, id(self)) def submitBuildRequest(self, req): @@ -316,6 +324,64 @@ self.building = [] self.slaves = [] + def consumeTheSoulOfYourPredecessor(self, old): + """Suck the brain out of an old Builder. + + This takes all the runtime state from an existing Builder and moves + it into ourselves. This is used when a Builder is changed in the + master.cfg file: the new Builder has a different factory, but we want + all the builds that were queued for the old one to get processed by + the new one. Any builds which are already running will keep running. + The new Builder will get as many of the old SlaveBuilder objects as + it wants.""" + + log.msg("consumeTheSoulOfYourPredecessor: %s feeding upon %s" % + (self, old)) + # we claim all the pending builds, removing them from the old + # Builder's queue. This insures that the old Builder will not start + # any new work. + log.msg(" stealing %s buildrequests" % len(old.buildable)) + self.buildable.extend(old.buildable) + old.buildable = [] + + # old.building is not migrated: it keeps track of builds which were + # in progress in the old Builder. When those builds finish, the old + # Builder will be notified, not us. However, since the old + # SlaveBuilder will point to us, it is our maybeStartBuild() that + # will be triggered. + if old.building: + self.builder_status.setBigState("building") + + # Our set of slavenames may be different. Steal any of the old + # buildslaves that we want to keep using. + for sb in old.slaves[:]: + if sb.slave.slavename in self.slavenames: + log.msg(" stealing buildslave %s" % sb) + self.slaves.append(sb) + old.slaves.remove(sb) + sb.setBuilder(self) + + # old.attaching_slaves: + # these SlaveBuilders are waiting on a sequence of calls: + # remote.setMaster and remote.print . When these two complete, + # old._attached will be fired, which will add a 'connect' event to + # the builder_status and try to start a build. However, we've pulled + # everything out of the old builder's queue, so it will have no work + # to do. The outstanding remote.setMaster/print call will be holding + # the last reference to the old builder, so it will disappear just + # after that response comes back. + # + # The BotMaster will ask the slave to re-set their list of Builders + # shortly after this function returns, which will cause our + # attached() method to be fired with a bunch of references to remote + # SlaveBuilders, some of which we already have (by stealing them + # from the old Builder), some of which will be new. The new ones + # will be re-attached. + + # Therefore, we don't need to do anything about old.attaching_slaves + + return # all done + def fireTestEvent(self, name, with=None): if with is None: with = self @@ -360,7 +426,8 @@ # re-vivifies sb return defer.succeed(self) - sb = SlaveBuilder(self) + sb = SlaveBuilder() + sb.setBuilder(self) self.attaching_slaves.append(sb) d = sb.attached(slave, remote, commands) d.addCallback(self._attached) @@ -370,7 +437,6 @@ def _attached(self, sb): # TODO: make this .addSlaveEvent(slave.slavename, ['connect']) ? self.builder_status.addPointEvent(['connect', sb.slave.slavename]) - sb.state = IDLE self.attaching_slaves.remove(sb) self.slaves.append(sb) self.maybeStartBuild() @@ -433,13 +499,14 @@ self.fireTestEvent('idle') def maybeStartBuild(self): - log.msg("maybeStartBuild: %s %s" % (self.buildable, self.slaves)) + log.msg("maybeStartBuild %s: %s %s" % + (self, self.buildable, self.slaves)) if not self.buildable: self.updateBigStatus() return # nothing to do # find the first idle slave for sb in self.slaves: - if sb.state == IDLE: + if sb.isAvailable(): break else: log.msg("%s: want to start build, but we don't have a remote" @@ -480,12 +547,6 @@ watch the Build as it runs. """ self.building.append(build) - - # claim the slave. TODO: consider moving changes to sb.state inside - # SlaveBuilder.. that would be cleaner. - sb.state = PINGING - sb.startBuild(build) - self.updateBigStatus() log.msg("starting build %s.. pinging the slave" % build) @@ -501,8 +562,10 @@ def _startBuild_1(self, res, build, sb): if not res: return self._startBuildFailed("slave ping failed", build, sb) - # The buildslave is ready to go. - sb.state = BUILDING + # The buildslave is ready to go. sb.buildStarted() sets its state to + # BUILDING (so we won't try to use it for any other builds). This + # gets set back to IDLE by the Build itself when it finishes. + sb.buildStarted() d = sb.remote.callRemote("startBuild") d.addCallbacks(self._startBuild_2, self._startBuildFailed, callbackArgs=(build,sb), errbackArgs=(build,sb)) @@ -528,38 +591,30 @@ # put the build back on the buildable list log.msg("I tried to tell the slave that the build %s started, but " "remote_startBuild failed: %s" % (build, why)) - # release the slave - sb.finishBuild() - sb.state = IDLE + # release the slave. This will queue a call to maybeStartBuild, which + # will fire after other notifyOnDisconnect handlers have marked the + # slave as disconnected (so we don't try to use it again). + sb.buildFinished() log.msg("re-queueing the BuildRequest") self.building.remove(build) for req in build.requests: - self.buildable.insert(0, req) # they get first priority + self.buildable.insert(0, req) # the interrupted build gets first + # priority self.builder_status.addBuildRequest(req.status) - # other notifyOnDisconnect calls will mark the slave as disconnected. - # Re-try after they have fired, maybe there's another slave - # available. TODO: I don't like these un-synchronizable callLaters.. - # a better solution is to mark the SlaveBuilder as disconnected - # ourselves, but we'll need to make sure that they can tolerate - # multiple disconnects first. - reactor.callLater(0, self.maybeStartBuild) def buildFinished(self, build, sb): """This is called when the Build has finished (either success or failure). Any exceptions during the build are reported with results=FAILURE, not with an errback.""" - # release the slave - sb.finishBuild() - sb.state = IDLE - # otherwise the slave probably got removed in detach() + # by the time we get here, the Build has already released the slave + # (which queues a call to maybeStartBuild) self.building.remove(build) for req in build.requests: req.finished(build.build_status) - self.maybeStartBuild() def setExpectations(self, progress): """Mark the build as successful and update expectations for the next @@ -608,7 +663,7 @@ # see if there is an idle slave, so we can emit an appropriate error # message for sb in self.original.slaves: - if sb.state == IDLE: + if sb.isAvailable(): break else: if self.original.building: From warner at users.sourceforge.net Fri Nov 24 07:16:37 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:16:37 +0000 Subject: [Buildbot-commits] buildbot/buildbot/status builder.py,1.88,1.89 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/status In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv32609/buildbot/status Modified Files: builder.py Log Message: [project @ reconfig no longer interrupts builds, nor does it disconnect/reconnect slaves] Original author: warner at lothar.com Date: 2006-11-23 21:32:36 Index: builder.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/status/builder.py,v retrieving revision 1.88 retrieving revision 1.89 diff -u -d -r1.88 -r1.89 --- builder.py 15 Sep 2006 14:48:53 -0000 1.88 +++ builder.py 24 Nov 2006 07:16:35 -0000 1.89 @@ -1739,6 +1739,13 @@ def isConnected(self): return self.connected + def setAdmin(self, admin): + self.admin = admin + def setHost(self, host): + self.host = host + def setConnected(self, isConnected): + self.connected = isConnected + class Status: """ I represent the status of the buildmaster. From warner at users.sourceforge.net Fri Nov 24 07:16:38 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:16:38 +0000 Subject: [Buildbot-commits] buildbot/buildbot/test test_run.py, 1.40, 1.41 test_slaves.py, 1.4, 1.5 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/test In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv32609/buildbot/test Modified Files: test_run.py test_slaves.py Log Message: [project @ reconfig no longer interrupts builds, nor does it disconnect/reconnect slaves] Original author: warner at lothar.com Date: 2006-11-23 21:32:36 Index: test_run.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_run.py,v retrieving revision 1.40 retrieving revision 1.41 diff -u -d -r1.40 -r1.41 --- test_run.py 15 Sep 2006 14:47:41 -0000 1.40 +++ test_run.py 24 Nov 2006 07:16:35 -0000 1.41 @@ -497,10 +497,10 @@ def _testChangeBuilddir_2(self, res): bot = self.bot - # this causes the builder to be replaced - self.failIfIdentical(self.builder, bot.builders.get("dummy")) + # this does NOT cause the builder to be replaced builder = bot.builders.get("dummy") self.failUnless(builder) + self.failUnlessIdentical(self.builder, builder) # the basedir should be updated self.failUnlessEqual(builder.builddir, "dummy2") self.failUnlessEqual(builder.basedir, Index: test_slaves.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_slaves.py,v retrieving revision 1.4 retrieving revision 1.5 diff -u -d -r1.4 -r1.5 --- test_slaves.py 15 Sep 2006 14:47:41 -0000 1.4 +++ test_slaves.py 24 Nov 2006 07:16:35 -0000 1.5 @@ -3,11 +3,13 @@ from twisted.trial import unittest from buildbot.twcompat import maybeWait from twisted.internet import defer, reactor +from twisted.python import log from buildbot.test.runutils import RunMixin from buildbot.sourcestamp import SourceStamp from buildbot.process.base import BuildRequest from buildbot.status.builder import SUCCESS +from buildbot.slave import bot config_1 = """ from buildbot.process import factory @@ -21,12 +23,22 @@ c['slavePortnum'] = 0 c['schedulers'] = [] -f = factory.BuildFactory([s(dummy.RemoteDummy, timeout=1)]) +f1 = factory.BuildFactory([s(dummy.RemoteDummy, timeout=1)]) +f2 = factory.BuildFactory([s(dummy.RemoteDummy, timeout=2)]) c['builders'] = [ {'name': 'b1', 'slavenames': ['bot1','bot2','bot3'], - 'builddir': 'b1', 'factory': f}, + 'builddir': 'b1', 'factory': f1}, + ] +""" + +config_2 = config_1 + """ + +c['builders'] = [ + {'name': 'b1', 'slavenames': ['bot1','bot2','bot3'], + 'builddir': 'b1', 'factory': f2}, ] + """ class Slave(RunMixin, unittest.TestCase): @@ -167,6 +179,198 @@ def _testDontClaimPingingSlave_3(self, res): self.failUnlessEqual(res.getSlavename(), "bot1") +config_3 = """ +from buildbot.process import factory +from buildbot.steps import dummy +s = factory.s + +BuildmasterConfig = c = {} +c['bots'] = [('bot1', 'sekrit')] +c['sources'] = [] +c['schedulers'] = [] +c['slavePortnum'] = 0 +c['schedulers'] = [] + +f1 = factory.BuildFactory([s(dummy.Wait, handle='one')]) +f2 = factory.BuildFactory([s(dummy.Wait, handle='two')]) +f3 = factory.BuildFactory([s(dummy.Wait, handle='three')]) + +c['builders'] = [ + {'name': 'b1', 'slavenames': ['bot1'], + 'builddir': 'b1', 'factory': f1}, + ] +""" + +config_4 = config_3 + """ +c['builders'] = [ + {'name': 'b1', 'slavenames': ['bot1'], + 'builddir': 'b1', 'factory': f2}, + ] +""" + +config_5 = config_3 + """ +c['builders'] = [ + {'name': 'b1', 'slavenames': ['bot1'], + 'builddir': 'b1', 'factory': f3}, + ] +""" + +from buildbot.slave.commands import waitCommandRegistry + +class Reconfig(RunMixin, unittest.TestCase): + + def setUp(self): + RunMixin.setUp(self) + self.master.loadConfig(config_3) + self.master.startService() + d = self.connectSlave(["b1"]) + return maybeWait(d) + + def _one_started(self): + log.msg("testReconfig._one_started") + self.build1_started = True + self.d1.callback(None) + return self.d2 + + def _two_started(self): + log.msg("testReconfig._two_started") + self.build2_started = True + self.d3.callback(None) + return self.d4 + + def _three_started(self): + log.msg("testReconfig._three_started") + self.build3_started = True + self.d5.callback(None) + return self.d6 + + def testReconfig(self): + # reconfiguring a Builder should not interrupt any running Builds. No + # queued BuildRequests should be lost. The next Build started should + # use the new process. + slave1 = self.slaves['bot1'] + bot1 = slave1.getServiceNamed('bot') + sb1 = bot1.builders['b1'] + self.failUnless(isinstance(sb1, bot.SlaveBuilder)) + self.failUnless(sb1.running) + b1 = self.master.botmaster.builders['b1'] + self.orig_b1 = b1 + + self.d1 = d1 = defer.Deferred() + self.d2 = d2 = defer.Deferred() + self.d3, self.d4 = defer.Deferred(), defer.Deferred() + self.d5, self.d6 = defer.Deferred(), defer.Deferred() + self.build1_started = False + self.build2_started = False + self.build3_started = False + waitCommandRegistry[("one","build1")] = self._one_started + waitCommandRegistry[("two","build2")] = self._two_started + waitCommandRegistry[("three","build3")] = self._three_started + + # use different branches to make sure these cannot be merged + br1 = BuildRequest("build1", SourceStamp(branch="1")) + b1.submitBuildRequest(br1) + br2 = BuildRequest("build2", SourceStamp(branch="2")) + b1.submitBuildRequest(br2) + br3 = BuildRequest("build3", SourceStamp(branch="3")) + b1.submitBuildRequest(br3) + self.requests = (br1, br2, br3) + # all three are now in the queue + + # wait until the first one has started + d1.addCallback(self._testReconfig_2) + return d1 + + def _testReconfig_2(self, res): + log.msg("_testReconfig_2") + # confirm that it is building + brs = self.requests[0].status.getBuilds() + self.failUnlessEqual(len(brs), 1) + self.build1 = brs[0] + self.failUnlessEqual(self.build1.getCurrentStep().getName(), "wait") + # br1 is building, br2 and br3 are in the queue (in that order). Now + # we reconfigure the Builder. + self.failUnless(self.build1_started) + d = self.master.loadConfig(config_4) + d.addCallback(self._testReconfig_3) + return d + + def _testReconfig_3(self, res): + log.msg("_testReconfig_3") + # now check to see that br1 is still building, and that br2 and br3 + # are in the queue of the new builder + b1 = self.master.botmaster.builders['b1'] + self.failIfIdentical(b1, self.orig_b1) + self.failIf(self.build1.isFinished()) + self.failUnlessEqual(self.build1.getCurrentStep().getName(), "wait") + self.failUnlessEqual(len(b1.buildable), 2) + self.failUnless(self.requests[1] in b1.buildable) + self.failUnless(self.requests[2] in b1.buildable) + + # allow br1 to finish, and make sure its status is delivered normally + d = self.requests[0].waitUntilFinished() + d.addCallback(self._testReconfig_4) + self.d2.callback(None) + return d + + def _testReconfig_4(self, bs): + log.msg("_testReconfig_4") + self.failUnlessEqual(bs.getReason(), "build1") + self.failUnless(bs.isFinished()) + self.failUnlessEqual(bs.getResults(), SUCCESS) + + # at this point, the first build has finished, and there is a pending + # call to start the second build. Once that pending call fires, there + # is a network roundtrip before the 'wait' RemoteCommand is delivered + # to the slave. We need to wait for both events to happen before we + # can check to make sure it is using the correct process. Just wait a + # full second. + d = defer.Deferred() + d.addCallback(self._testReconfig_5) + reactor.callLater(1, d.callback, None) + return d + + def _testReconfig_5(self, res): + log.msg("_testReconfig_5") + # at this point the next build ought to be running + b1 = self.master.botmaster.builders['b1'] + self.failUnlessEqual(len(b1.buildable), 1) + self.failUnless(self.requests[2] in b1.buildable) + self.failUnlessEqual(len(b1.building), 1) + # and it ought to be using the new process + self.failUnless(self.build2_started) + + # now, while the second build is running, change the config multiple + # times. + + d = self.master.loadConfig(config_3) + d.addCallback(lambda res: self.master.loadConfig(config_4)) + d.addCallback(lambda res: self.master.loadConfig(config_5)) + def _done(res): + # then once that's done, allow the second build to finish and + # wait for it to complete + da = self.requests[1].waitUntilFinished() + self.d4.callback(None) + return da + d.addCallback(_done) + def _done2(res): + # and once *that*'s done, wait another second to let the third + # build start + db = defer.Deferred() + reactor.callLater(1, db.callback, None) + return db + d.addCallback(_done2) + d.addCallback(self._testReconfig_6) + return d + + def _testReconfig_6(self, res): + log.msg("_testReconfig_6") + # now check to see that the third build is running + self.failUnless(self.build3_started) + + # we're done + + class Slave2(RunMixin, unittest.TestCase): From warner at users.sourceforge.net Fri Nov 24 07:16:37 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:16:37 +0000 Subject: [Buildbot-commits] buildbot/buildbot/steps dummy.py,1.3,1.4 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/steps In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv32609/buildbot/steps Modified Files: dummy.py Log Message: [project @ reconfig no longer interrupts builds, nor does it disconnect/reconnect slaves] Original author: warner at lothar.com Date: 2006-11-23 21:32:36 Index: dummy.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/steps/dummy.py,v retrieving revision 1.3 retrieving revision 1.4 diff -u -d -r1.3 -r1.4 --- dummy.py 17 Sep 2006 20:35:49 -0000 1.3 +++ dummy.py 24 Nov 2006 07:16:35 -0000 1.4 @@ -4,6 +4,8 @@ from buildbot.process.buildstep import LoggedRemoteCommand from buildbot.status.builder import SUCCESS, FAILURE +# these classes are used internally by buildbot unit tests + class Dummy(BuildStep): """I am a dummy no-op step, which runs entirely on the master, and simply waits 5 seconds before finishing with SUCCESS @@ -79,3 +81,20 @@ cmd = LoggedRemoteCommand("dummy", args) self.startCommand(cmd) +class Wait(LoggingBuildStep): + """I start a command on the slave that waits for the unit test to + tell it when to finish. + """ + + name = "wait" + def __init__(self, handle, **kwargs): + LoggingBuildStep.__init__(self, **kwargs) + self.handle = handle + + def describe(self, done=False): + return ["wait: %s" % self.handle] + + def start(self): + args = {'handle': (self.handle, self.build.reason)} + cmd = LoggedRemoteCommand("dummy.wait", args) + self.startCommand(cmd) From warner at users.sourceforge.net Fri Nov 24 07:16:37 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:16:37 +0000 Subject: [Buildbot-commits] buildbot/buildbot/slave bot.py, 1.21, 1.22 commands.py, 1.70, 1.71 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/slave In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv32609/buildbot/slave Modified Files: bot.py commands.py Log Message: [project @ reconfig no longer interrupts builds, nor does it disconnect/reconnect slaves] Original author: warner at lothar.com Date: 2006-11-23 21:32:36 Index: bot.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/slave/bot.py,v retrieving revision 1.21 retrieving revision 1.22 diff -u -d -r1.21 -r1.22 --- bot.py 13 Oct 2006 09:08:25 -0000 1.21 +++ bot.py 24 Nov 2006 07:16:35 -0000 1.22 @@ -67,7 +67,7 @@ self.not_really = not_really def __repr__(self): - return "" % self.name + return "" % (self.name, id(self)) def setServiceParent(self, parent): service.Service.setServiceParent(self, parent) @@ -284,7 +284,9 @@ def remote_setBuilderList(self, wanted): retval = {} + wanted_dirs = [] for (name, builddir) in wanted: + wanted_dirs.append(builddir) b = self.builders.get(name, None) if b: if b.builddir != builddir: @@ -303,8 +305,15 @@ log.msg("removing old builder %s" % name) self.builders[name].disownServiceParent() del(self.builders[name]) + + for d in os.listdir(self.basedir): + if os.path.isdir(d): + if d not in wanted_dirs: + log.msg("I have a leftover directory '%s' that is not " + "being used by the buildmaster: you can delete " + "it now" % d) return retval - + def remote_print(self, message): log.msg("message from master:", message) Index: commands.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/slave/commands.py,v retrieving revision 1.70 retrieving revision 1.71 diff -u -d -r1.70 -r1.71 --- commands.py 5 Nov 2006 04:56:03 -0000 1.70 +++ commands.py 24 Nov 2006 07:16:35 -0000 1.71 @@ -970,7 +970,7 @@ class DummyCommand(Command): """ I am a dummy no-op command that by default takes 5 seconds to complete. - See L{buildbot.process.step.RemoteDummy} + See L{buildbot.steps.dummy.RemoteDummy} """ def start(self): @@ -1004,6 +1004,56 @@ registerSlaveCommand("dummy", DummyCommand, command_version) +# this maps handle names to a callable. When the WaitCommand starts, this +# callable is invoked with no arguments. It should return a Deferred. When +# that Deferred fires, our WaitCommand will finish. +waitCommandRegistry = {} + +class WaitCommand(Command): + """ + I am a dummy command used by the buildbot unit test suite. I want for the + unit test to tell us to finish. See L{buildbot.steps.dummy.Wait} + """ + + def start(self): + self.d = defer.Deferred() + log.msg(" starting wait command [%s]" % self.stepId) + handle = self.args['handle'] + cb = waitCommandRegistry[handle] + del waitCommandRegistry[handle] + def _called(): + log.msg(" wait-%s starting" % (handle,)) + d = cb() + def _done(res): + log.msg(" wait-%s finishing: %s" % (handle, res)) + return res + d.addBoth(_done) + d.addCallbacks(self.finished, self.failed) + reactor.callLater(0, _called) + return self.d + + def interrupt(self): + log.msg(" wait command interrupted") + if self.interrupted: + return + self.interrupted = True + self.finished("interrupted") + + def finished(self, res): + log.msg(" wait command finished [%s]" % self.stepId) + if self.interrupted: + self.sendStatus({'rc': 2}) + else: + self.sendStatus({'rc': 0}) + self.d.callback(0) + def failed(self, why): + log.msg(" wait command failed [%s]" % self.stepId) + self.sendStatus({'rc': 1}) + self.d.callback(0) + +registerSlaveCommand("dummy.wait", WaitCommand, command_version) + + class SourceBase(Command): """Abstract base class for Version Control System operations (checkout and update). This class extracts the following arguments from the From warner at users.sourceforge.net Fri Nov 24 07:16:45 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:16:45 +0000 Subject: [Buildbot-commits] buildbot/buildbot/process builder.py,1.38,1.39 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/process In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv32669/buildbot/process Modified Files: builder.py Log Message: [project @ Builder: avoid starting a build in the middle of reconfig] Original author: warner at lothar.com Date: 2006-11-24 06:22:00 Index: builder.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/process/builder.py,v retrieving revision 1.38 retrieving revision 1.39 diff -u -d -r1.38 -r1.39 --- builder.py 24 Nov 2006 07:16:35 -0000 1.38 +++ builder.py 24 Nov 2006 07:16:42 -0000 1.39 @@ -439,7 +439,7 @@ self.builder_status.addPointEvent(['connect', sb.slave.slavename]) self.attaching_slaves.remove(sb) self.slaves.append(sb) - self.maybeStartBuild() + reactor.callLater(0, self.maybeStartBuild) self.fireTestEvent('attach') return self From warner at users.sourceforge.net Fri Nov 24 07:16:44 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:16:44 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.776,1.777 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv32669 Modified Files: ChangeLog Log Message: [project @ Builder: avoid starting a build in the middle of reconfig] Original author: warner at lothar.com Date: 2006-11-24 06:22:00 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.776 retrieving revision 1.777 diff -u -d -r1.776 -r1.777 --- ChangeLog 24 Nov 2006 07:16:35 -0000 1.776 +++ ChangeLog 24 Nov 2006 07:16:42 -0000 1.777 @@ -1,3 +1,10 @@ +2006-11-24 Brian Warner + + * buildbot/process/builder.py (Builder._attached): delay the call + to maybeStartBuild for a reactor turn, to avoid starting a build + in the middle of a reconfig (say, if the new Builder uses a new + slave which is already connected). + 2006-11-23 Brian Warner * buildbot/master.py (BuildMaster.loadConfig_Builders): changing a From warner at users.sourceforge.net Fri Nov 24 07:16:50 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:16:50 +0000 Subject: [Buildbot-commits] buildbot/buildbot/status builder.py,1.89,1.90 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/status In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv32686/buildbot/status Modified Files: builder.py Log Message: [project @ status/builder.py: update comment] Original author: warner at lothar.com Date: 2006-11-24 06:54:19 Index: builder.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/status/builder.py,v retrieving revision 1.89 retrieving revision 1.90 diff -u -d -r1.89 -r1.90 --- builder.py 24 Nov 2006 07:16:35 -0000 1.89 +++ builder.py 24 Nov 2006 07:16:48 -0000 1.90 @@ -1508,7 +1508,8 @@ # remember the oldest-to-earliest flow here. "next" means earlier. - # TODO: interleave build steps and self.events by timestamp + # TODO: interleave build steps and self.events by timestamp. + # TODO: um, I think we're already doing that. eventIndex = -1 e = self.getEvent(eventIndex) From warner at users.sourceforge.net Fri Nov 24 07:19:35 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:35 +0000 Subject: [Buildbot-commits] buildbot/buildbot/test test_bonsaipoller.py, 1.1, 1.2 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/test In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1355/buildbot/test Modified Files: test_bonsaipoller.py Log Message: [project @ test_bonsaipoller.py: remove the 'import *' to appease pyflakes] Original author: warner at lothar.com Date: 2006-11-24 05:04:29 Index: test_bonsaipoller.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_bonsaipoller.py,v retrieving revision 1.1 retrieving revision 1.2 diff -u -d -r1.1 -r1.2 --- test_bonsaipoller.py 5 Nov 2006 05:13:14 -0000 1.1 +++ test_bonsaipoller.py 24 Nov 2006 07:19:33 -0000 1.2 @@ -1,7 +1,8 @@ # -*- test-case-name: buildbot.test.test_bonsaipoller -*- from twisted.trial import unittest -from buildbot.changes.bonsaipoller import * +from buildbot.changes.bonsaipoller import FileNode, CiNode, BonsaiResult, \ + BonsaiParser, BonsaiPoller, InvalidResultError, EmptyResult from StringIO import StringIO from copy import deepcopy From warner at users.sourceforge.net Fri Nov 24 07:19:35 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:35 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.777,1.778 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1355 Modified Files: ChangeLog Log Message: [project @ test_bonsaipoller.py: remove the 'import *' to appease pyflakes] Original author: warner at lothar.com Date: 2006-11-24 05:04:29 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.777 retrieving revision 1.778 diff -u -d -r1.777 -r1.778 --- ChangeLog 24 Nov 2006 07:16:42 -0000 1.777 +++ ChangeLog 24 Nov 2006 07:19:33 -0000 1.778 @@ -7,6 +7,9 @@ 2006-11-23 Brian Warner + * buildbot/test/test_bonsaipoller.py: remove the 'import *' that + keeps pyflakes from finding undefined names + * buildbot/master.py (BuildMaster.loadConfig_Builders): changing a Builder no longer induces a disconnect/reconnect cycle. This means that any builds currently in progress will not be interrupted, and From warner at users.sourceforge.net Fri Nov 24 07:19:42 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:42 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.778,1.779 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1379 Modified Files: ChangeLog Log Message: [project @ test_transfer.py,test_steps.py: appease pyflakes] Original author: warner at lothar.com Date: 2006-11-24 05:16:02 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.778 retrieving revision 1.779 diff -u -d -r1.778 -r1.779 --- ChangeLog 24 Nov 2006 07:19:33 -0000 1.778 +++ ChangeLog 24 Nov 2006 07:19:39 -0000 1.779 @@ -7,6 +7,9 @@ 2006-11-23 Brian Warner + * buildbot/test/test_transfer.py: appease pyflakes + * buildbot/test/test_steps.py: same + * buildbot/test/test_bonsaipoller.py: remove the 'import *' that keeps pyflakes from finding undefined names From warner at users.sourceforge.net Fri Nov 24 07:19:42 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:42 +0000 Subject: [Buildbot-commits] buildbot/buildbot/test test_steps.py, 1.33, 1.34 test_transfer.py, 1.2, 1.3 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/test In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1379/buildbot/test Modified Files: test_steps.py test_transfer.py Log Message: [project @ test_transfer.py,test_steps.py: appease pyflakes] Original author: warner at lothar.com Date: 2006-11-24 05:16:02 Index: test_steps.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_steps.py,v retrieving revision 1.33 retrieving revision 1.34 diff -u -d -r1.33 -r1.34 --- test_steps.py 5 Oct 2006 00:45:58 -0000 1.33 +++ test_steps.py 24 Nov 2006 07:19:40 -0000 1.34 @@ -364,6 +364,13 @@ from buildbot.process.step import FailingDummy from buildbot.process.step import RemoteDummy + # now trick pyflakes into thinking we care + unused = [LogObserver, LogLineObserver, RemoteShellCommand, + BuildStep, LoggingBuildStep, ShellCommand, WithProperties, + TreeSize, Configure, Compile, Test, CVS, SVN, Darcs, + Git, Arch, Bazaar, Mercurial, P4, P4Sync, + Dummy, FailingDummy, RemoteDummy] + class _SimpleBuildStep(buildstep.BuildStep): def start(self): Index: test_transfer.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_transfer.py,v retrieving revision 1.2 retrieving revision 1.3 diff -u -d -r1.2 -r1.3 --- test_transfer.py 15 Sep 2006 14:49:46 -0000 1.2 +++ test_transfer.py 24 Nov 2006 07:19:40 -0000 1.3 @@ -2,7 +2,6 @@ import os from twisted.trial import unittest -from twisted.internet import defer from buildbot.twcompat import maybeWait from buildbot.steps.transfer import FileUpload, FileDownload from buildbot.test.runutils import StepTester From warner at users.sourceforge.net Fri Nov 24 07:19:50 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:50 +0000 Subject: [Buildbot-commits] buildbot/buildbot/clients debug.glade, 1.2, 1.3 debug.py, 1.4, 1.5 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/clients In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1419/buildbot/clients Modified Files: debug.glade debug.py Log Message: [project @ debugclient: add Ping Builder, update Request Build button] Original author: warner at lothar.com Date: 2006-11-24 06:24:12 Index: debug.glade =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/clients/debug.glade,v retrieving revision 1.2 retrieving revision 1.3 diff -u -d -r1.2 -r1.3 --- debug.glade 25 Oct 2005 01:57:33 -0000 1.2 +++ debug.glade 24 Nov 2006 07:19:48 -0000 1.3 @@ -18,6 +18,7 @@ GDK_WINDOW_TYPE_HINT_NORMAL GDK_GRAVITY_NORTH_WEST True + False @@ -146,6 +147,110 @@ + + True + False + 0 + + + + True + True + Branch: + True + GTK_RELIEF_NORMAL + True + False + False + True + + + + 0 + False + False + + + + + + True + True + True + True + 0 + + True + * + False + + + 0 + True + True + + + + + 0 + True + True + + + + + + True + False + 0 + + + + True + True + Revision: + True + GTK_RELIEF_NORMAL + True + False + False + True + + + + 0 + False + False + + + + + + True + True + True + True + 0 + + True + * + False + + + 0 + True + True + + + + + 0 + True + True + + + + 4 True @@ -276,110 +381,6 @@ True - - - - True - False - 0 - - - - True - True - Branch: - True - GTK_RELIEF_NORMAL - True - False - False - True - - - - 0 - False - False - - - - - - True - True - True - True - 0 - - True - * - False - - - 0 - True - True - - - - - 0 - True - True - - - - - - True - False - 0 - - - - True - True - Revision: - True - GTK_RELIEF_NORMAL - True - False - False - True - - - - 0 - False - False - - - - - - True - True - True - True - 0 - - True - * - False - - - 0 - True - True - - - - - 0 - True - True - - @@ -496,7 +497,7 @@ True True - Force + Request Build True GTK_RELIEF_NORMAL @@ -511,7 +512,21 @@ - + + True + True + Ping +Builder + True + GTK_RELIEF_NORMAL + True + + + + 0 + False + False + Index: debug.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/clients/debug.py,v retrieving revision 1.4 retrieving revision 1.5 diff -u -d -r1.4 -r1.5 --- debug.py 6 Sep 2006 00:41:55 -0000 1.4 +++ debug.py 24 Nov 2006 07:19:48 -0000 1.5 @@ -37,6 +37,7 @@ c('do_rebuild', self.do_rebuild) c('do_poke_irc', self.do_poke_irc) c('do_build', self.do_build) + c('do_ping', self.do_ping) c('do_commit', self.do_commit) c('on_usebranch_toggled', self.usebranch_toggled) self.usebranch_toggled(g('usebranch')) @@ -93,7 +94,26 @@ if not self.remote: return name = self.buildname.get_text() - d = self.remote.callRemote("forceBuild", name) + branch = None + if self.xml.get_widget("usebranch").get_active(): + branch = self.xml.get_widget('branch').get_text() + if branch == '': + branch = None + revision = None + if self.xml.get_widget("userevision").get_active(): + revision = self.xml.get_widget('revision').get_text() + if revision == '': + revision = None + reason = "debugclient 'Request Build' button pushed" + d = self.remote.callRemote("requestBuild", + name, reason, branch, revision) + d.addErrback(self.err) + + def do_ping(self, widget): + if not self.remote: + return + name = self.buildname.get_text() + d = self.remote.callRemote("pingBuilder", name) d.addErrback(self.err) def usebranch_toggled(self, widget): From warner at users.sourceforge.net Fri Nov 24 07:19:50 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:50 +0000 Subject: [Buildbot-commits] buildbot/buildbot master.py,1.98,1.99 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1419/buildbot Modified Files: master.py Log Message: [project @ debugclient: add Ping Builder, update Request Build button] Original author: warner at lothar.com Date: 2006-11-24 06:24:12 Index: master.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/master.py,v retrieving revision 1.98 retrieving revision 1.99 diff -u -d -r1.98 -r1.99 --- master.py 24 Nov 2006 07:16:35 -0000 1.98 +++ master.py 24 Nov 2006 07:19:48 -0000 1.99 @@ -24,8 +24,10 @@ from buildbot.util import now from buildbot.pbutil import NewCredPerspective from buildbot.process.builder import Builder, IDLE +from buildbot.process.base import BuildRequest from buildbot.status.builder import SlaveStatus, Status from buildbot.changes.changes import Change, ChangeMaster +from buildbot.sourcestamp import SourceStamp from buildbot import interfaces ######################################## @@ -398,10 +400,17 @@ def detached(self, mind): pass - def perspective_forceBuild(self, buildername, who=None): + def perspective_requestBuild(self, buildername, reason, branch, revision): c = interfaces.IControl(self.master) bc = c.getBuilder(buildername) - bc.forceBuild(who, "debug tool 'Force Build' button pushed") + ss = SourceStamp(branch, revision) + br = BuildRequest(reason, ss, buildername) + bc.requestBuild(br) + + def perspective_pingBuilder(self, buildername): + c = interfaces.IControl(self.master) + bc = c.getBuilder(buildername) + bc.ping() def perspective_fakeChange(self, file, revision=None, who="fakeUser", branch=None): From warner at users.sourceforge.net Fri Nov 24 07:19:50 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:50 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.779,1.780 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1419 Modified Files: ChangeLog Log Message: [project @ debugclient: add Ping Builder, update Request Build button] Original author: warner at lothar.com Date: 2006-11-24 06:24:12 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.779 retrieving revision 1.780 diff -u -d -r1.779 -r1.780 --- ChangeLog 24 Nov 2006 07:19:39 -0000 1.779 +++ ChangeLog 24 Nov 2006 07:19:47 -0000 1.780 @@ -1,5 +1,12 @@ 2006-11-24 Brian Warner + * buildbot/clients/debug.py: replace 'Force Build' button with + 'Request Build' (which just adds one to the queue), add Ping + Builder, add branch/revision fields to Request Build. + * buildbot/clients/debug.glade: same + * buildbot/master.py: update interface to match. This creates an + incompatibility between new debugclients and old buildmasters. + * buildbot/process/builder.py (Builder._attached): delay the call to maybeStartBuild for a reactor turn, to avoid starting a build in the middle of a reconfig (say, if the new Builder uses a new From warner at users.sourceforge.net Fri Nov 24 07:19:57 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:57 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.780,1.781 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1465 Modified Files: ChangeLog Log Message: [project @ finally remove forceBuild] Original author: warner at lothar.com Date: 2006-11-24 06:54:31 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.780 retrieving revision 1.781 diff -u -d -r1.780 -r1.781 --- ChangeLog 24 Nov 2006 07:19:47 -0000 1.780 +++ ChangeLog 24 Nov 2006 07:19:55 -0000 1.781 @@ -1,5 +1,14 @@ 2006-11-24 Brian Warner + * buildbot/interfaces.py (IBuilderControl.forceBuild): remove this + method, it has been deprecated for a long time. Use + IBuilderControl.requestBuild instead. + * buildbot/process/builder.py (BuilderControl.forceBuild): remove + * buildbot/master.py (BotPerspective.perspective_forceBuild): same + * buildbot/slave/bot.py (Bot.debug_forceBuild): same + * buildbot/test/test_control.py (Force.testForce): same + * buildbot/test/test_run.py: use requestBuild instead + * buildbot/clients/debug.py: replace 'Force Build' button with 'Request Build' (which just adds one to the queue), add Ping Builder, add branch/revision fields to Request Build. From warner at users.sourceforge.net Fri Nov 24 07:19:57 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:57 +0000 Subject: [Buildbot-commits] buildbot/buildbot/test test_control.py, 1.12, 1.13 test_run.py, 1.41, 1.42 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/test In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1465/buildbot/test Modified Files: test_control.py test_run.py Log Message: [project @ finally remove forceBuild] Original author: warner at lothar.com Date: 2006-11-24 06:54:31 Index: test_control.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_control.py,v retrieving revision 1.12 retrieving revision 1.13 diff -u -d -r1.12 -r1.13 --- test_control.py 15 Sep 2006 14:47:41 -0000 1.12 +++ test_control.py 24 Nov 2006 07:19:55 -0000 1.13 @@ -72,51 +72,34 @@ dl.append(defer.maybeDeferred(self.master.stopService)) return maybeWait(defer.DeferredList(dl)) - def testForce(self): - # TODO: since BuilderControl.forceBuild has been deprecated, this - # test is scheduled to be removed soon + def testRequest(self): m = self.master m.loadConfig(config) m.startService() d = self.connectSlave() - d.addCallback(self._testForce_1) + d.addCallback(self._testRequest_1) return maybeWait(d) - - def _testForce_1(self, res): + def _testRequest_1(self, res): c = interfaces.IControl(self.master) + req = base.BuildRequest("I was bored", SourceStamp()) builder_control = c.getBuilder("force") - d = builder_control.forceBuild("bob", "I was bored") - d.addCallback(self._testForce_2) + d = defer.Deferred() + req.subscribe(d.callback) + builder_control.requestBuild(req) + d.addCallback(self._testRequest_2) + # we use the same check-the-results code as testForce return d - def _testForce_2(self, build_control): + def _testRequest_2(self, build_control): self.failUnless(providedBy(build_control, interfaces.IBuildControl)) d = build_control.getStatus().waitUntilFinished() - d.addCallback(self._testForce_3) + d.addCallback(self._testRequest_3) return d - def _testForce_3(self, bs): + def _testRequest_3(self, bs): self.failUnless(providedBy(bs, interfaces.IBuildStatus)) self.failUnless(bs.isFinished()) self.failUnlessEqual(bs.getResults(), SUCCESS) #self.failUnlessEqual(bs.getResponsibleUsers(), ["bob"]) # TODO self.failUnlessEqual(bs.getChanges(), []) #self.failUnlessEqual(bs.getReason(), "forced") # TODO - - def testRequest(self): - m = self.master - m.loadConfig(config) - m.startService() - d = self.connectSlave() - d.addCallback(self._testRequest_1) - return maybeWait(d) - def _testRequest_1(self, res): - c = interfaces.IControl(self.master) - req = base.BuildRequest("I was bored", SourceStamp()) - builder_control = c.getBuilder("force") - d = defer.Deferred() - req.subscribe(d.callback) - builder_control.requestBuild(req) - d.addCallback(self._testForce_2) - # we use the same check-the-results code as testForce - return d Index: test_run.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/test/test_run.py,v retrieving revision 1.41 retrieving revision 1.42 diff -u -d -r1.41 -r1.42 --- test_run.py 24 Nov 2006 07:16:35 -0000 1.41 +++ test_run.py 24 Nov 2006 07:19:55 -0000 1.42 @@ -174,19 +174,16 @@ self.failUnlessEqual(bs.getResults(), builder.FAILURE) - - def testIdle1(self): - # disconnect the slave before the build starts - d = self.shutdownAllSlaves() # dies before it gets started - d.addCallback(self._testIdle1_1) + def submitBuild(self): + ss = SourceStamp() + br = BuildRequest("forced build", ss, "dummy") + self.control.getBuilder("dummy").requestBuild(br) + d = defer.Deferred() + def _started(bc): + br.unsubscribe(_started) + d.callback(bc) + br.subscribe(_started) return d - def _testIdle1_1(self, res): - # trying to force a build now will cause an error. Regular builds - # just wait for the slave to re-appear, but forced builds that - # cannot be run right away trigger NoSlaveErrors - fb = self.control.getBuilder("dummy").forceBuild - self.failUnlessRaises(interfaces.NoSlaveError, - fb, None, "forced build") def testIdle2(self): # now suppose the slave goes missing @@ -220,7 +217,7 @@ # least 3 seconds to complete, and this batch of commands must # complete within that time. # - d = self.control.getBuilder("dummy").forceBuild(None, "forced build") + d = self.submitBuild() d.addCallback(self._testBuild1_1) return maybeWait(d) @@ -249,7 +246,7 @@ def testBuild2(self): # this next sequence is timing-dependent - d = self.control.getBuilder("dummy").forceBuild(None, "forced build") + d = self.submitBuild() d.addCallback(self._testBuild1_1) return maybeWait(d, 30) testBuild2.timeout = 30 @@ -279,7 +276,7 @@ def testBuild3(self): # this next sequence is timing-dependent - d = self.control.getBuilder("dummy").forceBuild(None, "forced build") + d = self.submitBuild() d.addCallback(self._testBuild3_1) return maybeWait(d, 30) testBuild3.timeout = 30 @@ -306,7 +303,7 @@ def testBuild4(self): # this next sequence is timing-dependent - d = self.control.getBuilder("dummy").forceBuild(None, "forced build") + d = self.submitBuild() d.addCallback(self._testBuild4_1) return maybeWait(d, 30) testBuild4.timeout = 30 @@ -331,7 +328,7 @@ def testInterrupt(self): # this next sequence is timing-dependent - d = self.control.getBuilder("dummy").forceBuild(None, "forced build") + d = self.submitBuild() d.addCallback(self._testInterrupt_1) return maybeWait(d, 30) testInterrupt.timeout = 30 From warner at users.sourceforge.net Fri Nov 24 07:19:57 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:57 +0000 Subject: [Buildbot-commits] buildbot/buildbot/slave bot.py,1.22,1.23 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/slave In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1465/buildbot/slave Modified Files: bot.py Log Message: [project @ finally remove forceBuild] Original author: warner at lothar.com Date: 2006-11-24 06:54:31 Index: bot.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/slave/bot.py,v retrieving revision 1.22 retrieving revision 1.23 diff -u -d -r1.22 -r1.23 --- bot.py 24 Nov 2006 07:16:35 -0000 1.22 +++ bot.py 24 Nov 2006 07:19:55 -0000 1.23 @@ -335,10 +335,6 @@ files[f] = open(filename, "r").read() return files - def debug_forceBuild(self, name): - d = self.perspective.callRemote("forceBuild", name) - d.addCallbacks(log.msg, log.err) - class BotFactory(ReconnectingPBClientFactory): # 'keepaliveInterval' serves two purposes. The first is to keep the # connection alive: it guarantees that there will be at least some From warner at users.sourceforge.net Fri Nov 24 07:19:57 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:57 +0000 Subject: [Buildbot-commits] buildbot/buildbot/process builder.py,1.39,1.40 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/process In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1465/buildbot/process Modified Files: builder.py Log Message: [project @ finally remove forceBuild] Original author: warner at lothar.com Date: 2006-11-24 06:54:31 Index: builder.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/process/builder.py,v retrieving revision 1.39 retrieving revision 1.40 diff -u -d -r1.39 -r1.40 --- builder.py 24 Nov 2006 07:16:42 -0000 1.39 +++ builder.py 24 Nov 2006 07:19:55 -0000 1.40 @@ -643,52 +643,6 @@ else: __implements__ = interfaces.IBuilderControl, - def forceBuild(self, who, reason): - """This is a shortcut for building the current HEAD. - - (false: You get back a BuildRequest, just as if you'd asked politely. - To get control of the resulting build, you'll need use - req.subscribe() .) - - (true: You get back a Deferred that fires with an IBuildControl) - - This shortcut peeks into the Builder and raises an exception if there - is no slave available, to make backwards-compatibility a little - easier. - """ - - warnings.warn("Please use BuilderControl.requestBuildSoon instead", - category=DeprecationWarning, stacklevel=1) - - # see if there is an idle slave, so we can emit an appropriate error - # message - for sb in self.original.slaves: - if sb.isAvailable(): - break - else: - if self.original.building: - raise interfaces.BuilderInUseError("All slaves are in use") - raise interfaces.NoSlaveError("There are no slaves connected") - - req = base.BuildRequest(reason, sourcestamp.SourceStamp()) - self.requestBuild(req) - # this is a hack that fires the Deferred for the first build and - # ignores any others - class Watcher: - def __init__(self, req): - self.req = req - def wait(self): - self.d = d = defer.Deferred() - req.subscribe(self.started) - return d - def started(self, bs): - if self.d: - self.req.unsubscribe(self.started) - self.d.callback(bs) - self.d = None - w = Watcher(req) - return w.wait() - def requestBuild(self, req): """Submit a BuildRequest to this Builder.""" self.original.submitBuildRequest(req) From warner at users.sourceforge.net Fri Nov 24 07:19:57 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 07:19:57 +0000 Subject: [Buildbot-commits] buildbot/buildbot interfaces.py, 1.48, 1.49 master.py, 1.99, 1.100 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv1465/buildbot Modified Files: interfaces.py master.py Log Message: [project @ finally remove forceBuild] Original author: warner at lothar.com Date: 2006-11-24 06:54:31 Index: interfaces.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/interfaces.py,v retrieving revision 1.48 retrieving revision 1.49 diff -u -d -r1.48 -r1.49 --- interfaces.py 15 Oct 2006 17:51:11 -0000 1.48 +++ interfaces.py 24 Nov 2006 07:19:55 -0000 1.49 @@ -826,26 +826,6 @@ """Retrieve the IBuilderControl object for the given Builder.""" class IBuilderControl(Interface): - def forceBuild(who, reason): - """DEPRECATED, please use L{requestBuild} instead. - - Start a build of the latest sources. If 'who' is not None, it is - string with the name of the user who is responsible for starting the - build: they will be added to the 'interested users' list (so they may - be notified via email or another Status object when it finishes). - 'reason' is a string describing why this user requested the build. - - The results of forced builds are always sent to the Interested Users, - even if the Status object would normally only send results upon - failures. - - forceBuild() may raise L{NoSlaveError} or L{BuilderInUseError} if it - cannot start the build. - - forceBuild() returns a Deferred which fires with an L{IBuildControl} - object that can be used to further control the new build, or from - which an L{IBuildStatus} object can be obtained.""" - def requestBuild(request): """Queue a L{buildbot.process.base.BuildRequest} object for later building.""" Index: master.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/master.py,v retrieving revision 1.99 retrieving revision 1.100 diff -u -d -r1.99 -r1.100 --- master.py 24 Nov 2006 07:19:48 -0000 1.99 +++ master.py 24 Nov 2006 07:19:55 -0000 1.100 @@ -229,21 +229,6 @@ d.addCallbacks(_sent, _set_failed) return d - def perspective_forceBuild(self, name, who=None): - # slave admins are allowed to force any of their own builds - for b in self.builders: - if name == b.name: - try: - b.forceBuild(who, "slave requested build") - return "ok, starting build" - except interfaces.BuilderInUseError: - return "sorry, builder was in use" - except interfaces.NoSlaveError: - return "sorry, there is no slave to run the build" - else: - log.msg("slave requested build for unknown builder '%s'" % name) - return "sorry, invalid builder name" - def perspective_keepalive(self): pass @@ -1035,6 +1020,6 @@ components.registerAdapter(Control, BuildMaster, interfaces.IControl) -# so anybody who can get a handle on the BuildMaster can force a build with: -# IControl(master).getBuilder("full-2.3").forceBuild("me", "boredom") +# so anybody who can get a handle on the BuildMaster can cause a build with: +# IControl(master).getBuilder("full-2.3").requestBuild(buildrequest) From warner at users.sourceforge.net Fri Nov 24 08:23:29 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 08:23:29 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.781,1.782 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv27286 Modified Files: ChangeLog Log Message: [project @ enhance 'buildbot sighup' to show all related twistd.log lines. Also rename it to 'buildbot reconfig'. This addresses half of SF#1517975] Original author: warner at lothar.com Date: 2006-11-24 08:21:21 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.781 retrieving revision 1.782 diff -u -d -r1.781 -r1.782 --- ChangeLog 24 Nov 2006 07:19:55 -0000 1.781 +++ ChangeLog 24 Nov 2006 08:23:27 -0000 1.782 @@ -1,5 +1,18 @@ 2006-11-24 Brian Warner + * buildbot/scripts/runner.py (Options.subCommands): rename + 'buildbot sighup DIR' to 'buildbot reconfig DIR', but keep + 'sighup' as an alias. + * buildbot/scripts/reconfig.py (Reconfigurator): enhance the + reconfig command to follow twistd.log and print all of the lines + from the start of the config-file reload to its completion. This + should make it a lot easier to discover bugs in the config file. + Use --quiet to disable this behavior. This addresses half of + SF#1517975, the other half will be to add this same utility to + 'buildbot start' and 'buildbot restart'. + * docs/buildbot.texinfo (Loading the Config File): same + (Shutdown): same + * buildbot/interfaces.py (IBuilderControl.forceBuild): remove this method, it has been deprecated for a long time. Use IBuilderControl.requestBuild instead. From warner at users.sourceforge.net Fri Nov 24 08:23:29 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 08:23:29 +0000 Subject: [Buildbot-commits] buildbot/buildbot master.py,1.100,1.101 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv27286/buildbot Modified Files: master.py Log Message: [project @ enhance 'buildbot sighup' to show all related twistd.log lines. Also rename it to 'buildbot reconfig'. This addresses half of SF#1517975] Original author: warner at lothar.com Date: 2006-11-24 08:21:21 Index: master.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/master.py,v retrieving revision 1.100 retrieving revision 1.101 diff -u -d -r1.100 -r1.101 --- master.py 24 Nov 2006 07:19:55 -0000 1.100 +++ master.py 24 Nov 2006 08:23:27 -0000 1.101 @@ -625,6 +625,8 @@ except: log.msg("error during loadConfig") log.err() + log.msg("The new config file is unusable, so I'll ignore it.") + log.msg("I will keep using the previous config file instead.") f.close() def loadConfig(self, f): From warner at users.sourceforge.net Fri Nov 24 08:23:29 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 08:23:29 +0000 Subject: [Buildbot-commits] buildbot/buildbot/scripts reconfig.py, NONE, 1.1 runner.py, 1.45, 1.46 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/scripts In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv27286/buildbot/scripts Modified Files: runner.py Added Files: reconfig.py Log Message: [project @ enhance 'buildbot sighup' to show all related twistd.log lines. Also rename it to 'buildbot reconfig'. This addresses half of SF#1517975] Original author: warner at lothar.com Date: 2006-11-24 08:21:21 --- NEW FILE: reconfig.py --- import os, signal from twisted.internet import reactor, task from twisted.protocols.basic import LineOnlyReceiver class FakeTransport: disconnecting = False class LogWatcher(LineOnlyReceiver): POLL_INTERVAL = 0.1 delimiter = "\n" def __init__(self, finished): self.poller = task.LoopingCall(self.poll) self.in_reconfig = False self.finished_cb = finished self.transport = FakeTransport() def start(self, logfile): try: self.f = open(logfile, "rb") self.f.seek(0, 2) self.poller.start(self.POLL_INTERVAL) except IOError: print "Unable to follow %s" % logfile return False return True def finished(self, success): self.in_reconfig = False self.finished_cb(success) def lineReceived(self, line): if "loading configuration from" in line: self.in_reconfig = True if self.in_reconfig: print line if "I will keep using the previous config file" in line: self.finished(False) if "configuration update complete" in line: self.finished(True) def poll(self): while True: data = self.f.read(1000) if not data: return self.dataReceived(data) class Reconfigurator: def run(self, config): basedir = config['basedir'] quiet = config['quiet'] os.chdir(basedir) f = open("twistd.pid", "rt") pid = int(f.read().strip()) if quiet: os.kill(pid, signal.SIGHUP) return # keep reading twistd.log. Display all messages between "loading # configuration from ..." and "configuration update complete" or # "I will keep using the previous config file instead.", or until # 5 seconds have elapsed. reactor.callLater(5, self.timeout) self.lw = lw = LogWatcher(self.finished) if lw.start("twistd.log"): # we're watching # give the LogWatcher a chance to start reading print "sending SIGHUP to process %d" % pid reactor.callLater(0.2, os.kill, pid, signal.SIGHUP) reactor.run() else: # we couldn't watch the file.. just SIGHUP it os.kill(pid, signal.SIGHUP) print "sent SIGHUP to process %d" % pid def finished(self, success): if success: print "Reconfiguration is complete." else: print "Reconfiguration failed." reactor.stop() def timeout(self): print "Never saw reconfiguration finish." reactor.stop() def reconfig(config): r = Reconfigurator() r.run(config) Index: runner.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/scripts/runner.py,v retrieving revision 1.45 retrieving revision 1.46 diff -u -d -r1.45 -r1.46 --- runner.py 6 Sep 2006 00:41:55 -0000 1.45 +++ runner.py 24 Nov 2006 08:23:27 -0000 1.46 @@ -427,6 +427,15 @@ def getSynopsis(self): return "Usage: buildbot stop " +class ReconfigOptions(MakerBase): + optFlags = [ + ['quiet', 'q', "Don't display log messages about reconfiguration"], + ] + def getSynopsis(self): + return "Usage: buildbot reconfig " + + + class RestartOptions(MakerBase): def getSynopsis(self): return "Usage: buildbot restart " @@ -669,7 +678,9 @@ ['restart', None, RestartOptions, "Restart a buildmaster or buildslave"], - ['sighup', None, StopOptions, + ['reconfig', None, ReconfigOptions, + "SIGHUP a buildmaster to make it re-read the config file"], + ['sighup', None, ReconfigOptions, "SIGHUP a buildmaster to make it re-read the config file"], ['sendchange', None, SendChangeOptions, @@ -731,7 +742,8 @@ elif command == "restart": restart(so) elif command == "sighup": - stop(so, "HUP") + from buildbot.scripts.reconfig import Reconfigurator + Reconfigurator().run(so) elif command == "sendchange": sendchange(so, True) elif command == "debugclient": From warner at users.sourceforge.net Fri Nov 24 08:23:29 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 08:23:29 +0000 Subject: [Buildbot-commits] buildbot/docs buildbot.texinfo,1.85,1.86 Message-ID: Update of /cvsroot/buildbot/buildbot/docs In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv27286/docs Modified Files: buildbot.texinfo Log Message: [project @ enhance 'buildbot sighup' to show all related twistd.log lines. Also rename it to 'buildbot reconfig'. This addresses half of SF#1517975] Original author: warner at lothar.com Date: 2006-11-24 08:21:21 Index: buildbot.texinfo =================================================================== RCS file: /cvsroot/buildbot/buildbot/docs/buildbot.texinfo,v retrieving revision 1.85 retrieving revision 1.86 diff -u -d -r1.85 -r1.86 --- buildbot.texinfo 24 Nov 2006 07:16:36 -0000 1.85 +++ buildbot.texinfo 24 Nov 2006 08:23:27 -0000 1.86 @@ -1040,7 +1040,7 @@ config file. The following shortcut is available: @example -buildbot sighup @var{BASEDIR} +buildbot reconfig @var{BASEDIR} @end example When you update the Buildbot code to a new release, you will need to @@ -1894,9 +1894,13 @@ it: the @command{buildbot} tool has a shortcut for this: @example -buildbot sighup @var{BASEDIR} +buildbot reconfig @var{BASEDIR} @end example +This command will show you all of the lines from @file{twistd.log} +that relate to the reconfiguration. If there are any problems during +the config-file reload, they will be displayed in these lines. + The debug tool (@code{buildbot debugclient --master HOST:PORT}) has a ``Reload .cfg'' button which will also trigger a reload. In the future, there will be other ways to accomplish this step (probably a From warner at users.sourceforge.net Fri Nov 24 19:27:13 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 19:27:13 +0000 Subject: [Buildbot-commits] buildbot/docs buildbot.texinfo,1.86,1.87 Message-ID: Update of /cvsroot/buildbot/buildbot/docs In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv9198/docs Modified Files: buildbot.texinfo Log Message: [project @ buildbot statuslog: add reminders to connect to a PBListener port instead of the slaveport] Original author: warner at lothar.com Date: 2006-11-24 19:25:21 Index: buildbot.texinfo =================================================================== RCS file: /cvsroot/buildbot/buildbot/docs/buildbot.texinfo,v retrieving revision 1.86 retrieving revision 1.87 diff -u -d -r1.86 -r1.87 --- buildbot.texinfo 24 Nov 2006 08:23:27 -0000 1.86 +++ buildbot.texinfo 24 Nov 2006 19:27:11 -0000 1.87 @@ -5480,12 +5480,15 @@ prints out a new line each time an event occurs on the buildmaster. The @option{--master} option provides the location of the - at code{client.PBListener} status port, used to deliver build -information to realtime status clients. The option is always in the -form of a string, with hostname and port number separated by a colon -(@code{HOSTNAME:PORTNUM}). Note that this port is @emph{not} the same -as the slaveport (although a future version may allow the same port -number to be used for both purposes). + at code{buildbot.status.client.PBListener} status port, used to deliver +build information to realtime status clients. The option is always in +the form of a string, with hostname and port number separated by a +colon (@code{HOSTNAME:PORTNUM}). Note that this port is @emph{not} the +same as the slaveport (although a future version may allow the same +port number to be used for both purposes). If you get an error message +to the effect of ``Failure: twisted.cred.error.UnauthorizedLogin:'', +this may indicate that you are connecting to the slaveport rather than +a @code{PBListener} port. The @option{--master} option can also be provided by the @code{masterstatus} name in @file{.buildbot/options} (@pxref{.buildbot From warner at users.sourceforge.net Fri Nov 24 19:27:13 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 19:27:13 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.782,1.783 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv9198 Modified Files: ChangeLog Log Message: [project @ buildbot statuslog: add reminders to connect to a PBListener port instead of the slaveport] Original author: warner at lothar.com Date: 2006-11-24 19:25:21 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.782 retrieving revision 1.783 diff -u -d -r1.782 -r1.783 --- ChangeLog 24 Nov 2006 08:23:27 -0000 1.782 +++ ChangeLog 24 Nov 2006 19:27:10 -0000 1.783 @@ -1,5 +1,11 @@ 2006-11-24 Brian Warner + * buildbot/clients/base.py (TextClient.not_connected): upon + UnauthorizedLogin failures, remind the user to connect to the + PBListener port instead of the slaveport. + (TextClient.disconnected): shut down more quietly + * docs/buildbot.texinfo (statuslog): add another reminder + * buildbot/scripts/runner.py (Options.subCommands): rename 'buildbot sighup DIR' to 'buildbot reconfig DIR', but keep 'sighup' as an alias. From warner at users.sourceforge.net Fri Nov 24 19:27:13 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 19:27:13 +0000 Subject: [Buildbot-commits] buildbot/buildbot/clients base.py,1.12,1.13 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/clients In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv9198/buildbot/clients Modified Files: base.py Log Message: [project @ buildbot statuslog: add reminders to connect to a PBListener port instead of the slaveport] Original author: warner at lothar.com Date: 2006-11-24 19:25:21 Index: base.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/clients/base.py,v retrieving revision 1.12 retrieving revision 1.13 diff -u -d -r1.12 -r1.13 --- base.py 19 Jul 2005 23:12:01 -0000 1.12 +++ base.py 24 Nov 2006 19:27:11 -0000 1.13 @@ -3,7 +3,7 @@ import sys, re from twisted.spread import pb -from twisted.cred import credentials +from twisted.cred import credentials, error from twisted.internet import reactor class StatusClient(pb.Referenceable): @@ -93,15 +93,30 @@ creds = credentials.UsernamePassword("statusClient", "clientpw") d = cf.login(creds) reactor.connectTCP(host, port, cf) - d.addCallback(self.connected) + d.addCallbacks(self.connected, self.not_connected) return d def connected(self, ref): ref.notifyOnDisconnect(self.disconnected) self.listener.connected(ref) - + def not_connected(self, why): + if why.check(error.UnauthorizedLogin): + print """ +Unable to login.. are you sure we are connecting to a +buildbot.status.client.PBListener port and not to the slaveport? +""" + reactor.stop() + return why def disconnected(self, ref): print "lost connection" - reactor.stop() + # we can get here in one of two ways: the buildmaster has + # disconnected us (probably because it shut itself down), or because + # we've been SIGINT'ed. In the latter case, our reactor is already + # shut down, but we have no easy way of detecting that. So protect + # our attempt to shut down the reactor. + try: + reactor.stop() + except RuntimeError: + pass if __name__ == '__main__': master = "localhost:8007" From warner at users.sourceforge.net Fri Nov 24 20:02:49 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 20:02:49 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.783,1.784 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv23993 Modified Files: ChangeLog Log Message: [project @ buildbot reconfig: oops, forgot to enable it] Original author: warner at lothar.com Date: 2006-11-24 20:00:34 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.783 retrieving revision 1.784 diff -u -d -r1.783 -r1.784 --- ChangeLog 24 Nov 2006 19:27:10 -0000 1.783 +++ ChangeLog 24 Nov 2006 20:02:46 -0000 1.784 @@ -1,5 +1,8 @@ 2006-11-24 Brian Warner + * buildbot/scripts/runner.py (run): oops, forgot to enable the new + 'reconfig' command + * buildbot/clients/base.py (TextClient.not_connected): upon UnauthorizedLogin failures, remind the user to connect to the PBListener port instead of the slaveport. From warner at users.sourceforge.net Fri Nov 24 20:02:49 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 20:02:49 +0000 Subject: [Buildbot-commits] buildbot/buildbot/scripts runner.py,1.46,1.47 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/scripts In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv23993/buildbot/scripts Modified Files: runner.py Log Message: [project @ buildbot reconfig: oops, forgot to enable it] Original author: warner at lothar.com Date: 2006-11-24 20:00:34 Index: runner.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/scripts/runner.py,v retrieving revision 1.46 retrieving revision 1.47 diff -u -d -r1.46 -r1.47 --- runner.py 24 Nov 2006 08:23:27 -0000 1.46 +++ runner.py 24 Nov 2006 20:02:47 -0000 1.47 @@ -741,7 +741,7 @@ stop(so, wait=True) elif command == "restart": restart(so) - elif command == "sighup": + elif command == "reconfig" or command == "sighup": from buildbot.scripts.reconfig import Reconfigurator Reconfigurator().run(so) elif command == "sendchange": From warner at users.sourceforge.net Fri Nov 24 20:46:03 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 20:46:03 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.784,1.785 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv9870 Modified Files: ChangeLog Log Message: [project @ twisted_master.cfg: update to reflect current usage] Original author: warner at lothar.com Date: 2006-11-24 20:37:39 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.784 retrieving revision 1.785 diff -u -d -r1.784 -r1.785 --- ChangeLog 24 Nov 2006 20:02:46 -0000 1.784 +++ ChangeLog 24 Nov 2006 20:46:01 -0000 1.785 @@ -1,5 +1,8 @@ 2006-11-24 Brian Warner + * docs/examples/twisted_master.cfg: update to match the version + in use on twistedmatrix.com + * buildbot/scripts/runner.py (run): oops, forgot to enable the new 'reconfig' command From warner at users.sourceforge.net Fri Nov 24 20:46:03 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 20:46:03 +0000 Subject: [Buildbot-commits] buildbot/docs/examples twisted_master.cfg, 1.45, 1.46 Message-ID: Update of /cvsroot/buildbot/buildbot/docs/examples In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv9870/docs/examples Modified Files: twisted_master.cfg Log Message: [project @ twisted_master.cfg: update to reflect current usage] Original author: warner at lothar.com Date: 2006-11-24 20:37:39 Index: twisted_master.cfg =================================================================== RCS file: /cvsroot/buildbot/buildbot/docs/examples/twisted_master.cfg,v retrieving revision 1.45 retrieving revision 1.46 diff -u -d -r1.45 -r1.46 --- twisted_master.cfg 22 Sep 2006 05:57:58 -0000 1.45 +++ twisted_master.cfg 24 Nov 2006 20:46:01 -0000 1.46 @@ -11,19 +11,20 @@ import os.path -from buildbot import master from buildbot.changes.pb import PBChangeSource from buildbot.scheduler import Scheduler, Try_Userpass from buildbot.steps.source import SVN -from buildbot.steps.shell import ShellCommand from buildbot.process.factory import s from buildbot.process.process_twisted import \ QuickTwistedBuildFactory, \ FullTwistedBuildFactory, \ - TwistedDebsBuildFactory, \ TwistedReactorsBuildFactory from buildbot.status import html, words, client, mail +import extra_factory +reload(extra_factory) +from extra_factory import GoodTwistedBuildFactory + import private # holds passwords reload(private) # make it possible to change the contents without a restart @@ -100,8 +101,8 @@ "-Wignore::PendingDeprecationWarning:distutils.command.build_py", "-Wignore::PendingDeprecationWarning:distutils.command.build_ext", ] -b23 = {'name': "full-2.3", - 'slavename': "bot-exarkun-boson", +b23 = {'name': "debian-py2.3-select", + 'slavename': "bot-exarkun", 'builddir': "full2.3", 'factory': FullTwistedBuildFactory(source_copy, python=["python2.3", "-Wall"], @@ -112,7 +113,7 @@ } builders.append(b23) -b24 = {'name': "full-2.4", +b24 = {'name': "debian-py2.4-select", 'slavenames': ["bot-exarkun"], 'builddir': "full2.4", 'factory': FullTwistedBuildFactory(source_copy, @@ -123,59 +124,38 @@ } builders.append(b24) -bosx24 = {'name': 'osx-2.4', - 'slavenames': ['bot-exarkun-osx'], - 'builddir': 'full2.4-exarkun-osx', - 'factory': FullTwistedBuildFactory(source_copy, - python=["python2.4", "-Wall"], - # use -Werror soon - compileOpts=b24compile_opts, - runTestsRandomly=1), - } -builders.append(bosx24) - -bpollosx24 = {'name': 'poll-osx-2.4', - 'slavenames': ['bot-exarkun-osx-2'], - 'builddir': 'full2.4-poll-exarkun-osx', - 'factory': TwistedReactorsBuildFactory(source_copy, - python="python2.4", - reactors=['poll']), - } -builders.append(bpollosx24) +b24debian64 = { + 'name': 'debian64-py2.4-select', + 'slavenames': ['bot-idnar-debian64'], + 'builddir': 'full2.4-debian64', + 'factory': FullTwistedBuildFactory(source_copy, + python=["python2.4", "-Wall"], + compileOpts=b24compile_opts), + } +builders.append(b24debian64) b25debian = { - 'name': 'full-2.5-debian', + 'name': 'debian-py2.5-select', 'slavenames': ['bot-idnar-debian'], 'builddir': 'full2.5-debian', 'factory': FullTwistedBuildFactory(source_copy, python=["python2.5", "-Wall"], - compileOpts=b24compile_opts), - } + compileOpts=b24compile_opts)} builders.append(b25debian) -b24debian64 = { - 'name': 'full-2.4-debian64', - 'slavenames': ['bot-idnar-debian64'], - 'builddir': 'full2.4-debian64', +b25suse = { + 'name': 'suse-py2.5-select', + 'slavenames': ['bot-scmikes-2.5'], + 'builddir': 'bot-scmikes-2.5', 'factory': FullTwistedBuildFactory(source_copy, - python=["python2.4", "-Wall"], + python=["python2.5", "-Wall"], compileOpts=b24compile_opts), } -builders.append(b24debian64) - - -# debuild is offline while we figure out how to build 2.0 .debs from SVN -# b3 = {'name': "debuild", -# 'slavename': "bot2", -# 'builddir': "debuild", -# 'factory': TwistedDebsBuildFactory(source_export, -# python="python2.4"), -# } -# builders.append(b3) +builders.append(b25suse) -reactors = ['gtk2', 'gtk', 'qt', 'poll'] -b4 = {'name': "reactors", +reactors = ['poll', 'epoll', 'gtk', 'gtk2'] +b4 = {'name': "debian-py2.4-reactors", 'slavename': "bot2", 'builddir': "reactors", 'factory': TwistedReactorsBuildFactory(source_copy, @@ -184,54 +164,36 @@ } builders.append(b4) -# jml's machine is specifically for testing Qt -bqt = {'name': "Qt", - 'slavename': "bot-jml-qt", - 'builddir': "qt", - 'factory': TwistedReactorsBuildFactory(source_copy, - python="python2.4", - reactors=['qt']), - } -#builders.append(bqt) +bosx24 = { + 'name': 'osx-py2.4-select', + 'slavenames': ['bot-exarkun-osx'], + 'builddir': 'full2.4-exarkun-osx', + 'factory': FullTwistedBuildFactory(source_copy, + python=["python2.4", "-Wall"], + compileOpts=b24compile_opts, + runTestsRandomly=1)} +builders.append(bosx24) -jf = TwistedReactorsBuildFactory(source_copy, - python="python2.4", reactors=["default"]) -jf.steps.insert(0, s(ShellCommand, workdir=".", - command=["ktrace", "rm", "-rf", "Twisted"])) -b24osx = {'name': "OS-X", - 'slavename': "bot-jerub", - 'builddir': "OSX-full2.4", -# 'factory': TwistedReactorsBuildFactory(source_copy, -# python="python2.4", -# reactors=["default"], -# ), - 'factory': jf, - } -builders.append(b24osx) +forcegc = { + 'name': 'osx-py2.4-select-gc', + 'slavenames': ['bot-exarkun-osx'], + 'builddir': 'full2.4-force-gc-exarkun-osx', + 'factory': GoodTwistedBuildFactory(source_copy, + python="python2.4")} +builders.append(forcegc) -b24w32_select = { - 'name': "win32-select", - 'slavename': "bot-win32-select", - 'builddir': "W32-full2.4-select", - 'factory': TwistedReactorsBuildFactory(source_copy, - python="python", - compileOpts2=["-c","mingw32"], - reactors=["default"]), - } -#builders.append(b24w32_select) -b25suse = { - 'name': 'full-2.5-suse', - 'slavenames': ['bot-scmikes-2.5'], - 'builddir': 'bot-scmikes-2.5', - 'factory': FullTwistedBuildFactory(source_copy, - python=["python2.5", "-Wall"], - compileOpts=b24compile_opts), - } -builders.append(b25suse) +# debuild is offline while we figure out how to build 2.0 .debs from SVN +# b3 = {'name': "debuild", +# 'slavename': "bot2", +# 'builddir': "debuild", +# 'factory': TwistedDebsBuildFactory(source_export, +# python="python2.4"), +# } +# builders.append(b3) b24w32_scmikes_select = { - 'name': "win32-scmikes-select", + 'name': "win32-py2.4-select", 'slavename': "bot-scmikes-win32", 'builddir': "W32-full2.4-scmikes-select", 'factory': TwistedReactorsBuildFactory(source_copy, @@ -241,8 +203,19 @@ } builders.append(b24w32_scmikes_select) +b25w32_scmikes_select = { + 'name': "win32-py2.5-select", + 'slavename': "bot-scmikes-win32-2.5", + 'builddir': "W32-full2.5-scmikes-select", + 'factory': TwistedReactorsBuildFactory(source_copy, + python="python", + compileOpts2=["-c","mingw32"], + reactors=["default"]), + } +builders.append(b25w32_scmikes_select) + b24w32_win32er = { - 'name': "win32-win32er", + 'name': "win32-py2.4-er", 'slavename': "bot-win32-win32er", 'builddir': "W32-full2.4-win32er", 'factory': TwistedReactorsBuildFactory(source_copy, @@ -254,7 +227,7 @@ b24w32_iocp = { - 'name': "win32-iocp", + 'name': "win32-py2.4-iocp", 'slavename': "bot-win32-iocp", 'builddir': "W32-full2.4-iocp", 'factory': TwistedReactorsBuildFactory(source_copy, @@ -265,7 +238,7 @@ builders.append(b24w32_iocp) -b24freebsd = {'name': "freebsd", +b24freebsd = {'name': "freebsd-py2.4-select-kq", 'slavename': "bot-landonf", 'builddir': "freebsd-full2.4", 'factory': @@ -277,22 +250,25 @@ } builders.append(b24freebsd) -bpypyc = {'name': 'pypy-c', + +osxtsr = {'name': "osx-py2.4-tsr", + 'slavename': "bot-exarkun-osx", + 'builddir': "osx-tsr", + 'factory': TwistedReactorsBuildFactory( + source_copy, + python="python2.4", + reactors=["tsr"])} +builders.append(osxtsr) + + +bpypyc = {'name': 'osx-pypyc-select', 'slavename': 'bot-jerub-pypy', 'builddir': 'pypy-c', 'factory': TwistedReactorsBuildFactory(source_copy, - python="pypy-c", - reactors=["default"])} + python="pypy-c", + reactors=["default"])} builders.append(bpypyc) -# b24threadless = {'name': 'threadless', -# 'slavename': 'bot-threadless', -# 'builddir': 'debian-threadless-2.4', -# 'factory': TwistedReactorsBuildFactory(source_copy, -# python='python', -# reactors=['default'])} -# builders.append(b24threadless) - c['builders'] = builders # now set up the schedulers. We do this after setting up c['builders'] so we @@ -304,12 +280,15 @@ ## configure the schedulers s_quick = Scheduler(name="quick", branch=None, treeStableTimer=30, builderNames=["quick"]) -s_all = Scheduler(name="all", branch=None, treeStableTimer=5*60, - builderNames=all_builders) s_try = Try_Userpass("try", all_builders, port=9989, userpass=private.try_users) -c['schedulers'] = [s_quick, s_all, s_try] +s_all = [] +for i, builderName in enumerate(all_builders): + s_all.append(Scheduler(name="all-" + builderName, + branch=None, builderNames=[builderName], + treeStableTimer=(5 * 60 + i * 30))) +c['schedulers'] = [s_quick, s_try] + s_all @@ -336,7 +315,7 @@ c['manhole'] = manhole.PasswordManhole(*private.manhole) c['status'].append(client.PBListener(9936)) m = mail.MailNotifier(fromaddr="buildbot at twistedmatrix.com", - builders=["quick", "full-2.3"], + builders=["quick", "debian-py2.3-select"], sendToInterestedUsers=True, extraRecipients=["warner at lothar.com"], mode="problem", From warner at users.sourceforge.net Fri Nov 24 20:46:10 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 20:46:10 +0000 Subject: [Buildbot-commits] buildbot/docs/examples twisted_master.cfg, 1.46, 1.47 Message-ID: Update of /cvsroot/buildbot/buildbot/docs/examples In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv10235/docs/examples Modified Files: twisted_master.cfg Log Message: [project @ twisted_master.cfg: re-enable IRC bot] Original author: warner at lothar.com Date: 2006-11-24 20:44:37 Index: twisted_master.cfg =================================================================== RCS file: /cvsroot/buildbot/buildbot/docs/examples/twisted_master.cfg,v retrieving revision 1.46 retrieving revision 1.47 diff -u -d -r1.46 -r1.47 --- twisted_master.cfg 24 Nov 2006 20:46:01 -0000 1.46 +++ twisted_master.cfg 24 Nov 2006 20:46:08 -0000 1.47 @@ -301,12 +301,10 @@ c['status'].append(html.Waterfall(distrib_port=p)) else: c['status'].append(html.Waterfall(http_port=9988)) -# IRC disabled until I figure out why it won't connect successfully -# -warner 22-Aug-2006 -#if really: -# c['status'].append(words.IRC(host="irc.us.freenode.net", -# nick='buildbot', -# channels=["twisted"])) +if really: + c['status'].append(words.IRC(host="irc.freenode.net", + nick='buildbot', + channels=["twisted"])) c['debugPassword'] = private.debugPassword #c['interlocks'] = [("do-deb", ["full-2.2"], ["debuild"])] From warner at users.sourceforge.net Fri Nov 24 20:46:10 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Fri, 24 Nov 2006 20:46:10 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.785,1.786 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv10235 Modified Files: ChangeLog Log Message: [project @ twisted_master.cfg: re-enable IRC bot] Original author: warner at lothar.com Date: 2006-11-24 20:44:37 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.785 retrieving revision 1.786 diff -u -d -r1.785 -r1.786 --- ChangeLog 24 Nov 2006 20:46:01 -0000 1.785 +++ ChangeLog 24 Nov 2006 20:46:08 -0000 1.786 @@ -2,6 +2,9 @@ * docs/examples/twisted_master.cfg: update to match the version in use on twistedmatrix.com + (IRC): re-enable IRC bot. The 'irc.us.freenode.net' hostname I + was using before stopped working, but 'irc.freenode.net' works + just fine. * buildbot/scripts/runner.py (run): oops, forgot to enable the new 'reconfig' command From warner at users.sourceforge.net Sat Nov 25 07:57:34 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sat, 25 Nov 2006 07:57:34 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.786,1.787 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv26991 Modified Files: ChangeLog Log Message: [project @ SF#1517975: enhance 'start' and 'restart' commands to emit the useful parts of twistd.log, up until the process has started properly] Original author: warner at lothar.com Date: 2006-11-25 07:55:52 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.786 retrieving revision 1.787 diff -u -d -r1.786 -r1.787 --- ChangeLog 24 Nov 2006 20:46:08 -0000 1.786 +++ ChangeLog 25 Nov 2006 07:57:31 -0000 1.787 @@ -1,3 +1,20 @@ +2006-11-25 Brian Warner + + * buildbot/scripts/runner.py: enhance 'start' and 'restart' to + follow twistd.log and print lines until the process has started + started properly. For the buildmaster, this means until the config + file has been parsed and accepted. For the buildslave, this means + until we've connected to the master. We give up after 5 seconds in + any case. Helpful error messages and troubleshooting suggestions + are printed when we don't see a successful startup. This closes the + remainder of SF#1517975. + * buildbot/scripts/startup.py: moved app startup code to here + * buildbot/scripts/logwatcher.py: utility class to follow log + * buildbot/scripts/reconfig.py: rewrite to use LogWatcher + * buildbot/slave/bot.py: announce our BuildSlaveness to the log + so the LogWatcher can tell the difference between a buildmaster + and a buildslave + 2006-11-24 Brian Warner * docs/examples/twisted_master.cfg: update to match the version From warner at users.sourceforge.net Sat Nov 25 07:57:35 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sat, 25 Nov 2006 07:57:35 +0000 Subject: [Buildbot-commits] buildbot/buildbot/slave bot.py,1.23,1.24 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/slave In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv26991/buildbot/slave Modified Files: bot.py Log Message: [project @ SF#1517975: enhance 'start' and 'restart' commands to emit the useful parts of twistd.log, up until the process has started properly] Original author: warner at lothar.com Date: 2006-11-25 07:55:52 Index: bot.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/slave/bot.py,v retrieving revision 1.23 retrieving revision 1.24 diff -u -d -r1.23 -r1.24 --- bot.py 24 Nov 2006 07:19:55 -0000 1.23 +++ bot.py 25 Nov 2006 07:57:32 -0000 1.24 @@ -462,6 +462,7 @@ def __init__(self, host, port, name, passwd, basedir, keepalive, usePTY, keepaliveTimeout=30, umask=None, debugOpts={}): + log.msg("Creating BuildSlave") service.MultiService.__init__(self) self.debugOpts = debugOpts.copy() bot = self.botClass(basedir, usePTY) From warner at users.sourceforge.net Sat Nov 25 07:57:34 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sat, 25 Nov 2006 07:57:34 +0000 Subject: [Buildbot-commits] buildbot/buildbot/scripts logwatcher.py, NONE, 1.1 startup.py, NONE, 1.1 reconfig.py, 1.1, 1.2 runner.py, 1.47, 1.48 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/scripts In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv26991/buildbot/scripts Modified Files: reconfig.py runner.py Added Files: logwatcher.py startup.py Log Message: [project @ SF#1517975: enhance 'start' and 'restart' commands to emit the useful parts of twistd.log, up until the process has started properly] Original author: warner at lothar.com Date: 2006-11-25 07:55:52 --- NEW FILE: logwatcher.py --- import os from twisted.python.failure import Failure from twisted.internet import task, defer, reactor from twisted.protocols.basic import LineOnlyReceiver class FakeTransport: disconnecting = False class BuildmasterTimeoutError(Exception): pass class BuildslaveTimeoutError(Exception): pass class ReconfigError(Exception): pass class BuildSlaveDetectedError(Exception): pass class LogWatcher(LineOnlyReceiver): POLL_INTERVAL = 0.1 TIMEOUT_DELAY = 5.0 delimiter = os.linesep def __init__(self, logfile): self.logfile = logfile self.in_reconfig = False self.transport = FakeTransport() self.f = None self.processtype = "buildmaster" def start(self): # return a Deferred that fires when the reconfig process has # finished. It errbacks with TimeoutError if the finish line has not # been seen within 5 seconds, and with ReconfigError if the error # line was seen. If the logfile could not be opened, it errbacks with # an IOError. self.running = True d = defer.maybeDeferred(self._start) return d def _start(self): self.d = defer.Deferred() try: self.f = open(self.logfile, "rb") self.f.seek(0, 2) # start watching from the end except IOError: pass reactor.callLater(self.TIMEOUT_DELAY, self.timeout) self.poller = task.LoopingCall(self.poll) self.poller.start(self.POLL_INTERVAL) return self.d def timeout(self): if self.processtype == "buildmaster": self.d.errback(BuildmasterTimeoutError()) else: self.d.errback(BuildslaveTimeoutError()) def finished(self, results): self.running = False self.in_reconfig = False self.d.callback(results) def lineReceived(self, line): if not self.running: return if "Log opened." in line: self.in_reconfig = True if "loading configuration from" in line: self.in_reconfig = True if "Creating BuildSlave" in line: self.processtype = "buildslave" if self.in_reconfig: print line if "message from master: attached" in line: return self.finished("buildslave") if "I will keep using the previous config file" in line: return self.finished(Failure(ReconfigError())) if "configuration update complete" in line: return self.finished("buildmaster") def poll(self): if not self.f: try: self.f = open(self.logfile, "rb") except IOError: return while True: data = self.f.read(1000) if not data: return self.dataReceived(data) Index: reconfig.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/scripts/reconfig.py,v retrieving revision 1.1 retrieving revision 1.2 diff -u -d -r1.1 -r1.2 --- reconfig.py 24 Nov 2006 08:23:27 -0000 1.1 +++ reconfig.py 25 Nov 2006 07:57:32 -0000 1.2 @@ -1,52 +1,9 @@ import os, signal -from twisted.internet import reactor, task -from twisted.protocols.basic import LineOnlyReceiver - -class FakeTransport: - disconnecting = False - -class LogWatcher(LineOnlyReceiver): - POLL_INTERVAL = 0.1 - delimiter = "\n" - - def __init__(self, finished): - self.poller = task.LoopingCall(self.poll) - self.in_reconfig = False - self.finished_cb = finished - self.transport = FakeTransport() - - - def start(self, logfile): - try: - self.f = open(logfile, "rb") - self.f.seek(0, 2) - self.poller.start(self.POLL_INTERVAL) - except IOError: - print "Unable to follow %s" % logfile - return False - return True - - def finished(self, success): - self.in_reconfig = False - self.finished_cb(success) - - def lineReceived(self, line): - if "loading configuration from" in line: - self.in_reconfig = True - if self.in_reconfig: - print line - if "I will keep using the previous config file" in line: - self.finished(False) - if "configuration update complete" in line: - self.finished(True) +from twisted.internet import reactor - def poll(self): - while True: - data = self.f.read(1000) - if not data: - return - self.dataReceived(data) +from buildbot.scripts.logwatcher import LogWatcher, BuildmasterTimeoutError, \ + ReconfigError class Reconfigurator: def run(self, config): @@ -55,36 +12,49 @@ quiet = config['quiet'] os.chdir(basedir) f = open("twistd.pid", "rt") - pid = int(f.read().strip()) + self.pid = int(f.read().strip()) if quiet: - os.kill(pid, signal.SIGHUP) + os.kill(self.pid, signal.SIGHUP) return + # keep reading twistd.log. Display all messages between "loading # configuration from ..." and "configuration update complete" or # "I will keep using the previous config file instead.", or until # 5 seconds have elapsed. - reactor.callLater(5, self.timeout) - self.lw = lw = LogWatcher(self.finished) - if lw.start("twistd.log"): - # we're watching - # give the LogWatcher a chance to start reading - print "sending SIGHUP to process %d" % pid - reactor.callLater(0.2, os.kill, pid, signal.SIGHUP) - reactor.run() - else: - # we couldn't watch the file.. just SIGHUP it - os.kill(pid, signal.SIGHUP) - print "sent SIGHUP to process %d" % pid - def finished(self, success): - if success: - print "Reconfiguration is complete." - else: - print "Reconfiguration failed." + self.sent_signal = False + lw = LogWatcher("twistd.log") + d = lw.start() + d.addCallbacks(self.success, self.failure) + reactor.callLater(0.2, self.sighup) + reactor.run() + + def sighup(self): + if self.sent_signal: + return + print "sending SIGHUP to process %d" % self.pid + self.sent_signal = True + os.kill(self.pid, signal.SIGHUP) + + def success(self, res): + print """ +Reconfiguration appears to have completed successfully. +""" reactor.stop() - def timeout(self): - print "Never saw reconfiguration finish." + def failure(self, why): + if why.check(BuildmasterTimeoutError): + print "Never saw reconfiguration finish." + elif why.check(ReconfigError): + print """ +Reconfiguration failed. Please inspect the master.cfg file for errors, +correct them, then try 'buildbot reconfig' again. +""" + elif why.check(IOError): + # we were probably unable to open the file in the first place + self.sighup() + else: + print "Error while following twistd.log: %s" % why reactor.stop() def reconfig(config): Index: runner.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/scripts/runner.py,v retrieving revision 1.47 retrieving revision 1.48 diff -u -d -r1.47 -r1.48 --- runner.py 24 Nov 2006 20:02:47 -0000 1.47 +++ runner.py 25 Nov 2006 07:57:32 -0000 1.48 @@ -290,41 +290,6 @@ if not m.quiet: print "buildslave configured in %s" % m.basedir -def start(config): - basedir = config['basedir'] - quiet = config['quiet'] - os.chdir(basedir) - sys.path.insert(0, os.path.abspath(os.getcwd())) - if os.path.exists("/usr/bin/make") and os.path.exists("Makefile.buildbot"): - # Preferring the Makefile lets slave admins do useful things like set - # up environment variables for the buildslave. - cmd = "make -f Makefile.buildbot start" - if not quiet: print cmd - os.system(cmd) - else: - # see if we can launch the application without actually having to - # spawn twistd, since spawning processes correctly is a real hassle - # on windows. - from twisted.python.runtime import platformType - argv = ["twistd", - "--no_save", - "--logfile=twistd.log", # windows doesn't use the same default - "--python=buildbot.tac"] - if platformType == "win32": - argv.append("--reactor=win32") - sys.argv = argv - - # this is copied from bin/twistd. twisted-1.3.0 uses twistw, while - # twisted-2.0.0 uses _twistw. - if platformType == "win32": - try: - from twisted.scripts._twistw import run - except ImportError: - from twisted.scripts.twistw import run - else: - from twisted.scripts.twistd import run - run() - def stop(config, signame="TERM", wait=False): import signal @@ -337,7 +302,8 @@ timer = 0 os.kill(pid, signum) if not wait: - print "sent SIG%s to process" % signame + if not quiet: + print "sent SIG%s to process" % signame return time.sleep(0.1) while timer < 5: @@ -345,19 +311,25 @@ try: os.kill(pid, 0) except OSError: - print "buildbot process %d is dead" % pid + if not quiet: + print "buildbot process %d is dead" % pid return timer += 1 time.sleep(1) - print "never saw process go away" + if not quiet: + print "never saw process go away" def restart(config): + quiet = config['quiet'] + from buildbot.scripts.startup import start stop(config, wait=True) - print "now restarting buildbot process.." + if not quiet: + print "now restarting buildbot process.." start(config) - # this next line might not be printed, if start() ended up running twistd - # inline - print "buildbot process has been restarted" + if not quiet: + # this next line might not be printed, if start() ended up running + # twistd inline + print "buildbot process has been restarted" def loadOptions(filename="options", here=None, home=None): @@ -420,6 +392,9 @@ return localDict class StartOptions(MakerBase): + optFlags = [ + ['quiet', 'q', "Don't display startup log messages"], + ] def getSynopsis(self): return "Usage: buildbot start " @@ -437,6 +412,9 @@ class RestartOptions(MakerBase): + optFlags = [ + ['quiet', 'q', "Don't display startup log messages"], + ] def getSynopsis(self): return "Usage: buildbot restart " @@ -736,6 +714,7 @@ elif command == "create-slave": createSlave(so) elif command == "start": + from buildbot.scripts.startup import start start(so) elif command == "stop": stop(so, wait=True) --- NEW FILE: startup.py --- import os, sys, time class Follower: def follow(self): from twisted.internet import reactor from buildbot.scripts.reconfig import LogWatcher self.rc = 0 print "Following twistd.log until startup finished.." lw = LogWatcher("twistd.log") d = lw.start() d.addCallbacks(self._success, self._failure) reactor.run() return self.rc def _success(self, processtype): from twisted.internet import reactor print "The %s appears to have (re)started correctly." % processtype self.rc = 0 reactor.stop() def _failure(self, why): from twisted.internet import reactor from buildbot.scripts.logwatcher import BuildmasterTimeoutError, \ ReconfigError, BuildslaveTimeoutError if why.check(BuildmasterTimeoutError): print """ The buildmaster took more than 5 seconds to start, so we were unable to confirm that it started correctly. Please 'tail twistd.log' and look for a line that says 'configuration update complete' to verify correct startup. """ elif why.check(BuildslaveTimeoutError): print """ The buildslave took more than 5 seconds to start and/or connect to the buildmaster, so we were unable to confirm that it started and connected correctly. Please 'tail twistd.log' and look for a line that says 'message from master: attached' to verify correct startup. If you see a bunch of messages like 'will retry in 6 seconds', your buildslave might not have the correct hostname or portnumber for the buildmaster, or the buildmaster might not be running. If you see messages like 'Failure: twisted.cred.error.UnauthorizedLogin' then your buildslave might be using the wrong botname or password. Please correct these problems and then restart the buildslave. """ elif why.check(ReconfigError): print """ The buildmaster appears to have encountered an error in the master.cfg config file during startup. It is probably running with an empty configuration right now. Please inspect and fix master.cfg, then restart the buildmaster. """ elif why.check(BuildSlaveDetectedError): print """ Buildslave is starting up, not following logfile. """ else: print """ Unable to confirm that the buildmaster started correctly. You may need to stop it, fix the config file, and restart. """ print why self.rc = 1 reactor.stop() def start(config): os.chdir(config['basedir']) if config['quiet']: return launch(config) # fork a child to launch the daemon, while the parent process tails the # logfile if os.fork(): # this is the parent rc = Follower().follow() sys.exit(rc) # this is the child: give the logfile-watching parent a chance to start # watching it before we start the daemon time.sleep(0.2) launch(config) def launch(config): sys.path.insert(0, os.path.abspath(os.getcwd())) if os.path.exists("/usr/bin/make") and os.path.exists("Makefile.buildbot"): # Preferring the Makefile lets slave admins do useful things like set # up environment variables for the buildslave. cmd = "make -f Makefile.buildbot start" if not config['quiet']: print cmd os.system(cmd) else: # see if we can launch the application without actually having to # spawn twistd, since spawning processes correctly is a real hassle # on windows. from twisted.python.runtime import platformType argv = ["twistd", "--no_save", "--logfile=twistd.log", # windows doesn't use the same default "--python=buildbot.tac"] if platformType == "win32": argv.append("--reactor=win32") sys.argv = argv # this is copied from bin/twistd. twisted-1.3.0 uses twistw, while # twisted-2.0.0 uses _twistw. if platformType == "win32": try: from twisted.scripts._twistw import run except ImportError: from twisted.scripts.twistw import run else: from twisted.scripts.twistd import run run() From warner at users.sourceforge.net Sun Nov 26 07:31:47 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 26 Nov 2006 07:31:47 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.787,1.788 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6750 Modified Files: ChangeLog Log Message: [project @ buildbot restart: remove dubious message] Original author: warner at lothar.com Date: 2006-11-26 07:28:02 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.787 retrieving revision 1.788 diff -u -d -r1.787 -r1.788 --- ChangeLog 25 Nov 2006 07:57:31 -0000 1.787 +++ ChangeLog 26 Nov 2006 07:31:44 -0000 1.788 @@ -1,3 +1,9 @@ +2006-11-26 Brian Warner + + * buildbot/scripts/runner.py (restart): remove the old message + that got printed after the buildbot was restarted.. it most cases + it didn't get printed at the right time anyways + 2006-11-25 Brian Warner * buildbot/scripts/runner.py: enhance 'start' and 'restart' to From warner at users.sourceforge.net Sun Nov 26 07:31:47 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 26 Nov 2006 07:31:47 +0000 Subject: [Buildbot-commits] buildbot/buildbot/scripts runner.py,1.48,1.49 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/scripts In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv6750/buildbot/scripts Modified Files: runner.py Log Message: [project @ buildbot restart: remove dubious message] Original author: warner at lothar.com Date: 2006-11-26 07:28:02 Index: runner.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/scripts/runner.py,v retrieving revision 1.48 retrieving revision 1.49 diff -u -d -r1.48 -r1.49 --- runner.py 25 Nov 2006 07:57:32 -0000 1.48 +++ runner.py 26 Nov 2006 07:31:45 -0000 1.49 @@ -326,10 +326,6 @@ if not quiet: print "now restarting buildbot process.." start(config) - if not quiet: - # this next line might not be printed, if start() ended up running - # twistd inline - print "buildbot process has been restarted" def loadOptions(filename="options", here=None, home=None): From warner at users.sourceforge.net Sun Nov 26 07:36:52 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 26 Nov 2006 07:36:52 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.788,1.789 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv8743 Modified Files: ChangeLog Log Message: [project @ skipt the whole watch-the-logfile thing under windows] Original author: warner at lothar.com Date: 2006-11-26 07:29:34 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.788 retrieving revision 1.789 diff -u -d -r1.788 -r1.789 --- ChangeLog 26 Nov 2006 07:31:44 -0000 1.788 +++ ChangeLog 26 Nov 2006 07:36:49 -0000 1.789 @@ -1,5 +1,8 @@ 2006-11-26 Brian Warner + * buildbot/scripts/startup.py (start): skip the whole + watch-the-logfile thing under windows, since it needs os.fork() + * buildbot/scripts/runner.py (restart): remove the old message that got printed after the buildbot was restarted.. it most cases it didn't get printed at the right time anyways From warner at users.sourceforge.net Sun Nov 26 07:36:52 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 26 Nov 2006 07:36:52 +0000 Subject: [Buildbot-commits] buildbot/buildbot/scripts startup.py,1.1,1.2 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/scripts In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv8743/buildbot/scripts Modified Files: startup.py Log Message: [project @ skipt the whole watch-the-logfile thing under windows] Original author: warner at lothar.com Date: 2006-11-26 07:29:34 Index: startup.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/scripts/startup.py,v retrieving revision 1.1 retrieving revision 1.2 diff -u -d -r1.1 -r1.2 --- startup.py 25 Nov 2006 07:57:32 -0000 1.1 +++ startup.py 26 Nov 2006 07:36:50 -0000 1.2 @@ -67,6 +67,11 @@ if config['quiet']: return launch(config) + # we probably can't do this os.fork under windows + from twisted.python.runtime import platformType + if platformType == "win32": + return launch(config) + # fork a child to launch the daemon, while the parent process tails the # logfile if os.fork(): From warner at users.sourceforge.net Sun Nov 26 08:05:26 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 26 Nov 2006 08:05:26 +0000 Subject: [Buildbot-commits] buildbot/buildbot scheduler.py,1.21,1.22 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv20416/buildbot Modified Files: scheduler.py Log Message: [project @ scheduler.Nightly: improve docs slightly] Original author: warner at lothar.com Date: 2006-11-26 07:52:08 Index: scheduler.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/scheduler.py,v retrieving revision 1.21 retrieving revision 1.22 diff -u -d -r1.21 -r1.22 --- scheduler.py 30 Sep 2006 22:36:22 -0000 1.21 +++ scheduler.py 26 Nov 2006 08:05:24 -0000 1.22 @@ -389,7 +389,9 @@ month=12, dayOfMonth=24, hour=12, minute=0) For dayOfWeek and dayOfMonth, builds are triggered if the date matches - either of them. Month and day numbers start at 1, not zero. + either of them. All time values are compared against the tuple returned + by time.localtime(), so month and dayOfMonth numbers start at 1, not + zero. dayOfWeek=0 is Monday, dayOfWeek=6 is Sunday. """ compare_attrs = ('name', 'builderNames', From warner at users.sourceforge.net Sun Nov 26 08:05:26 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 26 Nov 2006 08:05:26 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.789,1.790 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv20416 Modified Files: ChangeLog Log Message: [project @ scheduler.Nightly: improve docs slightly] Original author: warner at lothar.com Date: 2006-11-26 07:52:08 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.789 retrieving revision 1.790 diff -u -d -r1.789 -r1.790 --- ChangeLog 26 Nov 2006 07:36:49 -0000 1.789 +++ ChangeLog 26 Nov 2006 08:05:24 -0000 1.790 @@ -1,5 +1,7 @@ 2006-11-26 Brian Warner + * buildbot/scheduler.py (Nightly): Improve docs slightly. + * buildbot/scripts/startup.py (start): skip the whole watch-the-logfile thing under windows, since it needs os.fork() From warner at users.sourceforge.net Sun Nov 26 08:31:39 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 26 Nov 2006 08:31:39 +0000 Subject: [Buildbot-commits] buildbot ChangeLog,1.790,1.791 Message-ID: Update of /cvsroot/buildbot/buildbot In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv30968 Modified Files: ChangeLog Log Message: [project @ ShellCommand: you can't use command=WithProperties(stuff): improve docs and add an assertion] Original author: warner at lothar.com Date: 2006-11-26 08:30:59 Index: ChangeLog =================================================================== RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v retrieving revision 1.790 retrieving revision 1.791 diff -u -d -r1.790 -r1.791 --- ChangeLog 26 Nov 2006 08:05:24 -0000 1.790 +++ ChangeLog 26 Nov 2006 08:31:36 -0000 1.791 @@ -1,5 +1,11 @@ 2006-11-26 Brian Warner + * docs/buildbot.texinfo (Build Properties): remind users that + WithProperties must appear in a command= list, not as a top-level + instance. + * buildbot/steps/shell.py (ShellCommand.start): and assert that + we're sending a list or a single string to the RemoteShellCommand + * buildbot/scheduler.py (Nightly): Improve docs slightly. * buildbot/scripts/startup.py (start): skip the whole From warner at users.sourceforge.net Sun Nov 26 08:31:39 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 26 Nov 2006 08:31:39 +0000 Subject: [Buildbot-commits] buildbot/buildbot/steps shell.py,1.3,1.4 Message-ID: Update of /cvsroot/buildbot/buildbot/buildbot/steps In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv30968/buildbot/steps Modified Files: shell.py Log Message: [project @ ShellCommand: you can't use command=WithProperties(stuff): improve docs and add an assertion] Original author: warner at lothar.com Date: 2006-11-26 08:30:59 Index: shell.py =================================================================== RCS file: /cvsroot/buildbot/buildbot/buildbot/steps/shell.py,v retrieving revision 1.3 retrieving revision 1.4 diff -u -d -r1.3 -r1.4 --- shell.py 17 Sep 2006 20:35:49 -0000 1.3 +++ shell.py 26 Nov 2006 08:31:37 -0000 1.4 @@ -203,6 +203,7 @@ # to set up an argv array, an environment, or extra logfiles= (like # the Source subclasses) can just skip straight to startCommand() command = self._interpolateProperties(self.command) + assert isinstance(command, (list, tuple, str)) # create the actual RemoteShellCommand instance now kwargs = self.remote_kwargs kwargs['command'] = command From warner at users.sourceforge.net Sun Nov 26 08:31:40 2006 From: warner at users.sourceforge.net (Brian Warner) Date: Sun, 26 Nov 2006 08:31:40 +0000 Subject: [Buildbot-commits] buildbot/docs buildbot.texinfo,1.87,1.88 Message-ID: Update of /cvsroot/buildbot/buildbot/docs In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv30968/docs Modified Files: buildbot.texinfo Log Message: [project @ ShellCommand: you can't use command=WithProperties(stuff): improve docs and add an assertion] Original author: warner at lothar.com Date: 2006-11-26 08:30:59 Index: buildbot.texinfo =================================================================== RCS file: /cvsroot/buildbot/buildbot/docs/buildbot.texinfo,v retrieving revision 1.87 retrieving revision 1.88 diff -u -d -r1.87 -r1.88 --- buildbot.texinfo 24 Nov 2006 19:27:11 -0000 1.87 +++ buildbot.texinfo 26 Nov 2006 08:31:37 -0000 1.88 @@ -3939,7 +3939,12 @@ @end example Don't forget the extra ``s'' after the closing parenthesis! This is -the cause of many confusing errors. +the cause of many confusing errors. Also note that you can only use +WithProperties in the list form of the command= definition. You cannot +currently use it in the (discouraged) @code{command="stuff"} +single-string form. However, you can use something like + at code{command=["/bin/sh", "-c", "stuff", WithProperties(stuff)]} to +use both shell expansion and WithProperties interpolation. Note that, like python, you can either do positional-argument interpolation @emph{or} keyword-argument interpolation, not both. Thus