[Buildbot-commits] buildbot/buildbot pbutil.py,1.8,1.9 master.py,1.60,1.61

Brian Warner warner at users.sourceforge.net
Fri Apr 22 21:29:21 UTC 2005


Update of /cvsroot/buildbot/buildbot/buildbot
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv15396/buildbot

Modified Files:
	pbutil.py master.py 
Log Message:
Revision: arch at buildbot.sf.net--2004/buildbot--dev--0--patch-69
Creator:  Brian Warner <warner at monolith.lothar.com>

fix propagation of changes to .builddir

2005-04-22  Brian Warner  <warner at lothar.com>

   * buildbot/test/test_run.py (RunMixin.shutdownSlave): remove the
   whendone= argument, just return the Deferred and let the caller do
   what they want with it.
   (Disconnect.testBuild1): wait for shutdownSlave
   (Basedir.testChangeBuilddir): new test to make sure changes to the
   builddir actually get propagated to the slave

   * buildbot/slave/bot.py (SlaveBuilder.setBuilddir): use an
   explicit method, rather than passing the builddir in __init__ .
   Make sure to update self.basedir too, this was broken before.
   (Bot.remote_setBuilderList): use b.setBuilddir for both new
   builders and for ones that have just had their builddir changed.
   (BotFactory): add a class-level .perspective attribute, so
   BuildSlave.waitUntilDisconnected won't get upset when the
   connection hasn't yet been established
   (BuildSlave.__init__): keep track of the bot.Bot instance, so
   tests can reach through it to inspect the SlaveBuilders

   * buildbot/process/base.py (Build.buildException): explain the
   log.err with a log.msg
   * buildbot/process/builder.py (Builder.startBuild): same
   (Builder._startBuildFailed): improve error message

   * buildbot/pbutil.py (RBCP.failedToGetPerspective): if the failure
   occurred because we lost the brand-new connection, retry instead
   of giving up. If not, it's probably an authorization failure, and
   it makes sense to stop trying. Make sure we log.msg the reason
   that we're log.err'ing the failure, otherwise test failures are
   really hard to figure out.

   * buildbot/master.py: change loadConfig() to return a Deferred
   that doesn't fire until the change has been fully implemented.
   This means any connected slaves have been updated with the new
   builddir. This change makes it easier to test the code which
   actually implements this builddir-updating.
   (BotPerspective.addBuilder): return Deferred
   (BotPerspective.removeBuilder): same
   (BotPerspective.attached): same
   (BotPerspective._attached): same. finish with remote_print before
   starting the getSlaveInfo, instead of doing them in parallel
   (BotPerspective.list_done): same
   (BotMaster.removeSlave): same. Fix the typo that meant we weren't
   actually calling slave.disconnect()
   (BotMaster.addBuilder): same
   (BotMaster.removeBuilder): same
   (BuildMaster.loadConfig): same
   (BuildMaster.loadConfig_Slaves): same
   (BuildMaster.loadConfig_Sources): same
   (BuildMaster.loadConfig_Builders): same
   (BuildMaster.loadConfig_status): same

   * buildbot/changes/changes.py (ChangeMaster.removeSource): return
   a Deferred that fires when the source is finally removed


Index: master.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/master.py,v
retrieving revision 1.60
retrieving revision 1.61
diff -u -d -r1.60 -r1.61
--- master.py	22 Apr 2005 03:55:52 -0000	1.60
+++ master.py	22 Apr 2005 21:29:19 -0000	1.61
@@ -52,16 +52,29 @@
         self.slave = None # a RemoteReference to the Bot, when connected
 
     def addBuilder(self, builder):
-        """Called to add a builder after the slave has connected."""
+        """Called to add a builder after the slave has connected.
+
+        @return: a Deferred that indicates when an attached slave has
+        accepted the new builder."""
+
         self.builders.append(builder)
         if self.slave:
-            self.sendBuilderList()
+            return self.sendBuilderList()
+        return defer.succeed(None)
 
     def removeBuilder(self, builder):
+        """Tell the slave that the given builder has been removed, allowing
+        it to discard the associated L{buildbot.slave.bot.SlaveBuilder}
+        object.
+
+        @return: a Deferred that fires when the slave has finished removing
+                 the SlaveBuilder
+        """
         self.builders.remove(builder)
         if self.slave:
             builder.detached()
-            self.sendBuilderList()
+            return self.sendBuilderList()
+        return defer.succeed(None)
 
     def __repr__(self):
         return "<BotPerspective '%s', builders: %s>" % \
@@ -69,9 +82,10 @@
                 string.join(map(lambda b: b.name, self.builders), ','))
 
     def attached(self, mind):
-        # this is called when the slave connects. It returns a Deferred that
-        # fires with a suitable pb.IPerspective to give to the slave (i.e.
-        # 'self')
+        """This is called when the slave connects.
+
+        @return: a Deferred that fires with a suitable pb.IPerspective to
+                 give to the slave (i.e. 'self')"""
 
         if self.slave:
             # uh-oh, we've got a duplicate slave. The most likely
@@ -91,10 +105,11 @@
             d.addCallback(lambda res: self._attached(mind))
             return d
 
-        self._attached(mind)
-        return defer.succeed(self)
+        return self._attached(mind)
 
     def disconnect(self):
+        if not self.slave:
+            return defer.succeed(None)
         log.msg("disconnecting old slave %s now" % self.slavename)
 
         # all kinds of teardown will happen as a result of
@@ -132,20 +147,25 @@
         return d
 
     def _attached(self, mind):
-        # We go through a sequence of calls, gathering information, then
-        # tell our Builders that they have a slave to work with.
+        """We go through a sequence of calls, gathering information, then
+        tell our Builders that they have a slave to work with.
+
+        @return: a Deferred that fires (with 'self') when our Builders are
+                 prepared to deal with the slave.
+        """
         self.slave = mind
-        self.slave.callRemote("print", "attached").addErrback(lambda why: 0)
+        d = self.slave.callRemote("print", "attached")
+        d.addErrback(lambda why: 0)
         self.slave_status.connected = True
         log.msg("bot attached")
 
         # TODO: there is a window here (while we're retrieving slaveinfo)
         # during which a disconnect or a duplicate-slave will be confusing
-        d = self.slave.callRemote("getSlaveInfo")
-        d.addCallback(self.got_info)
-        d.addErrback(self.infoUnavailable)
+        d.addCallback(lambda res: self.slave.callRemote("getSlaveInfo"))
+        d.addCallbacks(self.got_info, self.infoUnavailable)
         d.addCallback(self._attached2)
-        return self
+        d.addCallback(lambda res: self)
+        return d
 
     def got_info(self, info):
         log.msg("Got slaveinfo from '%s'" % self.slavename)
@@ -212,14 +232,17 @@
 
     def list_done(self, blist):
         # this could come back at weird times. be prepared to handle oddness
+        dl = []
         for name, remote in blist.items():
             for b in self.builders:
                 if b.name == name:
                     # if we sent the builders list because of a config
                     # change, the Builder might already be attached.
                     # Builder.attached will ignore us if this happens.
-                    b.attached(remote, self.slave_commands)
+                    d = b.attached(remote, self.slave_commands)
+                    dl.append(d)
                     continue
+        return defer.DeferredList(dl)
 
     def _listFailed(self, why):
         log.msg("BotPerspective._listFailed")
@@ -297,8 +320,9 @@
         self.slaves[slavename] = slave
 
     def removeSlave(self, slavename):
-        d = self.slaves[slavename].disconnect
+        d = self.slaves[slavename].disconnect()
         del self.slaves[slavename]
+        return d
 
     def getBuildernames(self):
         return self.builderNames
@@ -307,7 +331,12 @@
         """This is called by the setup code to define what builds should be
         performed. Each Builder object has a build slave that should host
         that build: the builds cannot be done until the right slave
-        connects."""
+        connects.
+
+        @return: a Deferred that fires when an attached slave has accepted
+                 the new builder.
+        """
+
         if self.debug: print "addBuilder", builder
         log.msg("Botmaster.addBuilder(%s)" % builder.name)
 
@@ -324,9 +353,15 @@
         self.checkInactiveInterlocks() # TODO?: do this in caller instead?
 
         slave = self.slaves[slavename]
-        slave.addBuilder(builder)
+        return slave.addBuilder(builder)
 
     def removeBuilder(self, builder):
+        """Stop using a Builder.
+        This removes the Builder from the list of active Builders.
+
+        @return: a Deferred that fires when an attached slave has finished
+                 removing the SlaveBuilder
+        """
         if self.debug: print "removeBuilder", builder
         log.msg("Botmaster.removeBuilder(%s)" % builder.name)
         b = self.builders[builder.name]
@@ -346,7 +381,8 @@
         self.builderNames.remove(builder.name)
         slave = self.slaves.get(builder.slavename)
         if slave:
-            slave.removeBuilder(builder)
+            return slave.removeBuilder(builder)
+        return defer.succeed(None)
 
     def addInterlock(self, interlock):
         """This is called by the setup code to create build interlocks:
@@ -660,6 +696,13 @@
         f.close()
 
     def loadConfig(self, f):
+        """Internal function to load a specific configuration file. Any
+        errors in the file will be signalled by raising an exception.
+
+        @return: a Deferred that will fire (with None) when the configuration
+        changes have been completed. This may involve a round-trip to each
+        buildslave that was involved."""
+
         localDict = {'basedir': os.path.expanduser(self.basedir)}
         try:
             exec f in localDict
@@ -757,6 +800,9 @@
         # now we're committed to implementing the new configuration, so do
         # it atomically
 
+        # asynchronous updates can add a Deferred to this list
+        dl = []
+
         self.projectName = projectName
         self.projectURL = projectURL
         self.buildbotURL = buildbotURL
@@ -764,7 +810,7 @@
         # self.bots: Disconnect any that were attached and removed from the
         # list. Update self.checker with the new list of passwords,
         # including debug/change/status.
-        self.loadConfig_Slaves(bots)
+        dl.append(self.loadConfig_Slaves(bots))
 
         # self.debugPassword
         if debugPassword:
@@ -779,25 +825,28 @@
         if manhole != self.manhole:
             # changing
             if self.manhole:
-                # TODO: disownServiceParent may return Deferred
-                self.manhole.disownServiceParent()
+                # disownServiceParent may return a Deferred
+                d = defer.maybeDeferred(self.manhole.disownServiceParent)
+                dl.append(d)
                 self.manhole = None
             if manhole:
                 self.manhole = manhole
                 manhole.setServiceParent(self)
 
-        self.loadConfig_Sources(sources)
+        dl.append(self.loadConfig_Sources(sources))
 
         # add/remove self.botmaster.builders to match builders. The
         # botmaster will handle startup/shutdown issues.
-        self.loadConfig_Builders(builders)
+        dl.append(self.loadConfig_Builders(builders))
 
-        self.loadConfig_status(status, irc, webPortnum, webPathname)
+        d = self.loadConfig_status(status, irc, webPortnum, webPathname)
+        dl.append(d)
 
         # self.slavePort
         if self.slavePortnum != slavePortnum:
             if self.slavePort:
-                self.slavePort.disownServiceParent()
+                d = defer.maybeDeferred(self.slavePort.disownServiceParent)
+                dl.append(d)
                 self.slavePort = None
             if slavePortnum is not None:
                 self.slavePort = internet.TCPServer(slavePortnum,
@@ -810,6 +859,7 @@
         self.loadConfig_Interlocks(interlocks)
         
         log.msg("configuration updated")
+        return defer.DeferredList(dl)
 
     def loadConfig_Slaves(self, bots):
         # set up the Checker with the names and passwords of all valid bots
@@ -822,25 +872,28 @@
         old = self.bots; oldnames = [name for name,pw in old]
         new = bots; newnames = [name for name,pw in new]
         # removeSlave will hang up on the old bot
-        [self.botmaster.removeSlave(name)
-         for name in oldnames if name not in newnames]
+        dl = [self.botmaster.removeSlave(name)
+              for name in oldnames if name not in newnames]
         [self.botmaster.addSlave(name)
          for name in newnames if name not in oldnames]
 
         # all done
         self.bots = bots
+        return defer.DeferredList(dl)
 
     def loadConfig_Sources(self, sources):
         # shut down any that were removed, start any that were added
         old = self.sources
         new = sources
-        [self.change_svc.removeSource(source)
-         for source in old if source not in new]
+        dl = [self.change_svc.removeSource(source)
+              for source in old if source not in new]
         [self.change_svc.addSource(source)
          for source in new if source not in old]
         self.sources = sources
+        return defer.DeferredList(dl)
 
     def loadConfig_Builders(self, newBuilders):
+        dl = []
         old = self.botmaster.getBuildernames()
         newNames = []
         newList = {}
@@ -859,7 +912,8 @@
         for old in self.botmaster.builders.values()[:]:
             if old.name not in newList.keys():
                 log.msg("removing old builder %s" % old.name)
-                self.botmaster.removeBuilder(old)
+                d = self.botmaster.removeBuilder(old)
+                dl.append(d)
                 # announce the change
                 self.status.builderRemoved(old.name)
 
@@ -873,7 +927,8 @@
                 log.msg("adding new builder %s" % name)
                 statusbag = self.status.builderAdded(name, basedir)
                 builder = Builder(data, statusbag)
-                self.botmaster.addBuilder(builder)
+                d = self.botmaster.addBuilder(builder)
+                dl.append(d)
             else:
                 diffs = old.compareToSetup(data)
                 if not diffs: # unchanged: leave it alone
@@ -888,14 +943,17 @@
                     # make a new statusbag
                     statusbag = old.builder_status
                     statusbag.saveYourself() # seems like a good idea
-                    self.botmaster.removeBuilder(old)
+                    d = self.botmaster.removeBuilder(old)
+                    dl.append(d)
                     builder = Builder(data, statusbag)
                     # point out that the builder was updated
                     statusbag.addPointEvent(["config", "updated"])
-                    self.botmaster.addBuilder(builder)
+                    d = self.botmaster.addBuilder(builder)
+                    dl.append(d)
         # now that everything is up-to-date, make sure the names are in the
         # desired order
         self.botmaster.builderNames = newNames
+        return defer.DeferredList(dl)
 
     def loadConfig_status(self, status,
                           irc=None, webPortnum=None, webPathname=None):
@@ -921,12 +979,14 @@
             status.append(Waterfall(distrib_port=webPathname))
 
         # here is where the real work happens
+        dl = []
 
         # remove old ones
         for s in self.statusTargets[:]:
             if not s in status:
                 log.msg("removing IStatusReceiver", s)
-                s.disownServiceParent()
+                d = defer.maybeDeferred(s.disownServiceParent)
+                dl.append(d)
                 self.statusTargets.remove(s)
         # add new ones
         for s in status:
@@ -935,6 +995,8 @@
                 s.setServiceParent(self)
                 self.statusTargets.append(s)
 
+        return defer.DeferredList(dl)
+
     def loadConfig_Interlocks(self, newInterlocks):
         newList = {}
         for interlockData in newInterlocks:

Index: pbutil.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/pbutil.py,v
retrieving revision 1.8
retrieving revision 1.9
diff -u -d -r1.8 -r1.9
--- pbutil.py	6 Dec 2004 07:36:34 -0000	1.8
+++ pbutil.py	22 Apr 2005 21:29:19 -0000	1.9
@@ -129,5 +129,15 @@
         pass
 
     def failedToGetPerspective(self, why):
+        """The login process failed, most likely because of an authorization
+        failure (bad password), but it is also possible that we lost the new
+        connection before we managed to send our credentials.
+        """
+        log.msg("ReconnectingPBClientFactory.failedToGetPerspective")
+        if why.check(pb.PBConnectionLost):
+            log.msg("we lost the brand-new connection")
+            # retrying might help here, let clientConnectionLost decide
+            return
+        # probably authorization
         self.stopTrying() # logging in harder won't help
         log.err(why)





More information about the Commits mailing list