[Buildbot-commits] buildbot/buildbot buildset.py,NONE,1.1 locks.py,NONE,1.1 sourcestamp.py,NONE,1.1 scheduler.py,NONE,1.1 interfaces.py,1.26,1.27 master.py,1.73,1.74
Brian Warner
warner at users.sourceforge.net
Tue Jul 19 23:12:02 UTC 2005
- Previous message (by thread): [Buildbot-commits] buildbot/docs .arch-inventory,1.2,1.3 Makefile,1.1,1.2 buildbot.texinfo,1.8,1.9 .cvsignore,1.1,1.2
- Next message (by thread): [Buildbot-commits] buildbot ChangeLog,1.466,1.467 Makefile,1.13,1.14
- Messages sorted by:
[ date ]
[ thread ]
[ subject ]
[ author ]
Update of /cvsroot/buildbot/buildbot/buildbot
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv17398/buildbot
Modified Files:
interfaces.py master.py
Added Files:
buildset.py locks.py sourcestamp.py scheduler.py
Log Message:
Revision: arch at buildbot.sf.net--2004/buildbot--dev--0--patch-239
Creator: Brian Warner <warner at monolith.lothar.com>
merge in build-on-branch code: Merged from warner at monolith.lothar.com--2005 (patch 0-18, 40-41)
Patches applied:
* warner at monolith.lothar.com--2005/buildbot--dev--0--patch-40
Merged from arch at buildbot.sf.net--2004 (patch 232-238)
* warner at monolith.lothar.com--2005/buildbot--dev--0--patch-41
Merged from local-usebranches (warner at monolith.lothar.com--2005/buildbot--usebranches--0( (patch 0-18)
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--base-0
tag of warner at monolith.lothar.com--2005/buildbot--dev--0--patch-38
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-1
rearrange build scheduling
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-2
replace ugly 4-tuple with a distinct SourceStamp class
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-3
document upcoming features, clean up CVS branch= argument
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-4
Merged from arch at buildbot.sf.net--2004 (patch 227-231), warner at monolith.lothar.com--2005 (patch 39)
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-5
implement per-Step Locks, add tests (which all fail)
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-6
implement scheduler.Dependent, add (failing) tests
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-7
make test_dependencies work
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-8
finish making Locks work, tests now pass
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-9
fix test failures when run against twisted >2.0.1
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-10
rename test_interlock.py to test_locks.py
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-11
add more Locks tests, add branch examples to manual
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-12
rewrite test_vc.py, create repositories in setUp rather than offline
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-13
make new tests work with twisted-1.3.0
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-14
implement/test build-on-branch for most VC systems
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-15
minor changes: test-case-name tags, init cleanup
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-16
Merged from arch at buildbot.sf.net--2004 (patch 232-233)
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-17
Merged from arch at buildbot.sf.net--2004 (patch 234-236)
* warner at monolith.lothar.com--2005/buildbot--usebranches--0--patch-18
Merged from arch at buildbot.sf.net--2004 (patch 237-238), warner at monolith.lothar.com--2005 (patch 40)
--- NEW FILE: buildset.py ---
from twisted.internet import defer
from buildbot.process import base
from buildbot.status import builder
class BuildSet:
"""I represent a set of potential Builds, all of the same source tree,
across a specified list of Builders. I can represent a build of a
specific version of the source tree (named by source.branch and
source.revision), or a build of a certain set of Changes
(source.changes=list)."""
def __init__(self, builderNames, source, reason=None):
"""
@param source: a L{SourceStamp}
"""
self.builderNames = builderNames
self.source = source
self.reason = reason
self.set_status = bss = builder.BuildSetStatus()
bss.setSourceStamp(source)
bss.setReason(reason)
self.successWatchers = []
self.finishedWatchers = []
self.failed = False
def waitUntilSuccess(self):
"""Return a Deferred that will fire (with an IBuildSetStatus) when we
know whether or not this BuildSet will be a complete success (all
builds succeeding). This means it will fire upon the first failing
build, or upon the last successful one."""
# TODO: make it safe to call this after the buildset has completed
d = defer.Deferred()
self.successWatchers.append(d)
return d
def waitUntilFinished(self):
"""Return a Deferred that will fire when all builds have finished."""
d = defer.Deferred()
self.finishedWatchers.append(d)
return d
def start(self, builders):
"""This is called by the BuildMaster to actually create and submit
the BuildRequests."""
self.requests = []
reqs = []
# create the requests
for b in builders:
req = base.BuildRequest(self.reason, self.source)
reqs.append((b, req))
self.requests.append(req)
d = req.waitUntilFinished()
d.addCallback(self.requestFinished, req)
# now submit them
self.status = {} # maps requests to BuildStatus
for b,req in reqs:
b.submitBuildRequest(req)
def requestFinished(self, buildstatus, req):
self.requests.remove(req)
self.status[req] = buildstatus
if buildstatus.getResults() == builder.FAILURE:
if not self.failed:
self.failed = True
self.set_status.setResults(builder.FAILURE)
self.notifySuccessWatchers()
if not self.requests:
self.set_status.setResults(builder.SUCCESS)
self.notifyFinishedWatchers()
def notifySuccessWatchers(self):
for d in self.successWatchers:
d.callback(self.set_status)
self.successWatchers = []
def notifyFinishedWatchers(self):
if not self.failed:
self.notifySuccessWatchers()
for d in self.finishedWatchers:
d.callback(self.set_status)
self.finishedWatchers = []
Index: interfaces.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/interfaces.py,v
retrieving revision 1.26
retrieving revision 1.27
diff -u -d -r1.26 -r1.27
--- interfaces.py 15 May 2005 23:43:56 -0000 1.26
+++ interfaces.py 19 Jul 2005 23:11:59 -0000 1.27
@@ -34,6 +34,28 @@
"""Should return a string which briefly describes this source. This
string will be displayed in an HTML status page."""
+class IScheduler(Interface):
+ """I watch for Changes in the source tree and decide when to trigger
+ Builds. I create BuildSet objects and submit them to the BuildMaster. I
+ am a service, and the BuildMaster is always my parent."""
+
+ def addChange(change):
+ """A Change has just been dispatched by one of the ChangeSources.
+ Each Scheduler will receive this Change. I may decide to start a
+ build as a result, or I might choose to ignore it."""
+
+class IUpstreamScheduler(Interface):
+ """This marks an IScheduler as being eligible for use as the 'upstream='
+ argument to a buildbot.scheduler.Dependent instance."""
+
+ def subscribeToSuccessfulBuilds(target):
+ """Request that the target callbable be invoked after every
+ successful buildset. The target will be called with a single
+ argument: the SourceStamp used by the successful builds."""
+
+class ISourceStamp(Interface):
+ pass
+
class IEmailSender(Interface):
"""I know how to send email, and can be used by other parts of the
Buildbot to contact developers."""
@@ -78,6 +100,33 @@
"""Unregister an IStatusReceiver. No further status messgaes will be
delivered."""
+class IBuildSetStatus(Interface):
+ """I represent a set of Builds, each run on a separate Builder but all
+ using the same source tree."""
+
+ def getSourceStamp():
+ pass
+ def getReason():
+ pass
+ def getChanges():
+ pass
+ def getResponsibleUsers():
+ pass
+ def getInterestedUsers():
+ pass
+ def getBuilds():
+ """Return a list of IBuildStatus objects that represent my
+ component Builds."""
+ def isFinished():
+ pass
+ def waitUntilFirstFailure():
+ pass
+ def waitUntilFinished():
+ pass
+ def getResults():
+ pass
+
+
class ISlaveStatus(Interface):
def getName():
"""Return the name of the build slave."""
@@ -91,26 +140,36 @@
def isConnected():
"""Return True if the slave is currently online, False if not."""
+class ISchedulerStatus(Interface):
+ def getName():
+ """Return the name of this Scheduler (a string)."""
+
+ def getPendingBuildsets():
+ """Return an IBuildSet for all BuildSets that are pending. These
+ BuildSets are waiting for their tree-stable-timers to expire."""
+
+
class IBuilderStatus(Interface):
def getName():
"""Return the name of this Builder (a string)."""
def getState():
- """Return a tuple (state, ETA, build=None) for this Builder. 'state'
- is the so-called 'big-status', indicating overall status (as opposed
- to which step is currently running). It is a string, one of
- 'offline', 'idle', 'waiting', 'interlocked', or 'building'. In the
- 'waiting' and 'building' states, 'ETA' may be a number indicating
- how long the builder expectes to be in that state (expressed as
- seconds from now). 'ETA' may be None if it cannot be estimated or
- the state does not have an ETA. In the 'building' state, 'build'
- will be an IBuildStatus object representing the current build."""
- # we could make 'build' valid for 'waiting' and 'interlocked' too
+ # TODO: this isn't nearly as meaningful as it used to be
+ """Return a tuple (state, build=None) for this Builder. 'state' is
+ the so-called 'big-status', indicating overall status (as opposed to
+ which step is currently running). It is a string, one of 'offline',
+ 'idle', or 'building'. In the 'building' state, 'build' will be an
+ IBuildStatus object representing the current build."""
def getSlave():
"""Return an ISlaveStatus object for the buildslave that is used by
this builder."""
+ def getPendingBuilds():
+ """Return an IBuildRequestStatus object for all upcoming builds
+ (those which are ready to go but which are waiting for a buildslave
+ to be available."""
+
def getCurrentBuild():
"""Return an IBuildStatus object for the current build in progress.
If the state is not 'building', this will be None."""
@@ -150,41 +209,55 @@
delivered."""
class IBuildStatus(Interface):
- """I represent the status of a single build, which may or may not be
- finished."""
+ """I represent the status of a single Build/BuildRequest. It could be
+ finished, in-progress, or not yet started."""
def getBuilder():
"""
- Return the BuilderStatus that ran this build.
+ Return the BuilderStatus that owns this build.
@rtype: implementor of L{IBuilderStatus}
"""
- def getNumber():
- """Within each builder, each Build has a number. Return it."""
+ def isStarted():
+ """Return a boolean. True means the build has started, False means it
+ is still in the pending queue."""
- def getPreviousBuild():
- """Convenience method. Returns None if the previous build is
- unavailable."""
+ def waitUntilStarted():
+ """Return a Deferred that will fire (with this IBuildStatus instance
+ as an argument) when the build starts. If the build has already
+ started, this deferred will fire right away."""
- def getSourceStamp():
- """Return a tuple of (revision, patch) which can be used to re-create
- the source tree that this build used. 'revision' is a string, the
- sort you would pass to 'cvs co -r REVISION'. 'patch' is either None,
- or a string which represents a patch that should be applied with
- 'patch -p0 < PATCH' from the directory created by the checkout
- operation.
+ def isFinished():
+ """Return a boolean. True means the build has finished, False means
+ it is still running."""
+
+ def waitUntilFinished():
+ """Return a Deferred that will fire when the build finishes. If the
+ build has already finished, this deferred will fire right away. The
+ callback is given this IBuildStatus instance as an argument."""
- This method will return None if the source information is no longer
- available."""
- # TODO: it should be possible to expire the patch but still remember
- # that the build was r123+something.
def getReason():
"""Return a string that indicates why the build was run. 'changes',
'forced', and 'periodic' are the most likely values. 'try' will be
added in the future."""
+ def getSourceStamp():
+ """Return a tuple of (branch, revision, patch) which can be used to
+ re-create the source tree that this build used. 'branch' is a string
+ with a VC-specific meaning, or None to indicate that the checkout
+ step used its default branch. 'revision' is a string, the sort you
+ would pass to 'cvs co -r REVISION'. 'patch' is either None, or a
+ (level, diff) tuple which represents a patch that should be applied
+ with 'patch -pLEVEL < DIFF' from the directory created by the
+ checkout operation.
+
+ This method will return None if the source information is no longer
+ available."""
+ # TODO: it should be possible to expire the patch but still remember
+ # that the build was r123+something.
+
def getChanges():
"""Return a list of Change objects which represent which source
changes went into the build."""
@@ -204,6 +277,15 @@
make the Changes that went into it (build sheriffs, code-domain
owners)."""
+ # once the build has started, the following methods become available
+
+ def getNumber():
+ """Within each builder, each Build has a number. Return it."""
+
+ def getPreviousBuild():
+ """Convenience method. Returns None if the previous build is
+ unavailable."""
+
def getSteps():
"""Return a list of IBuildStepStatus objects. For invariant builds
(those which always use the same set of Steps), this should always
@@ -217,15 +299,6 @@
(seconds since the epoch) when the Build started and finished. If
the build is still running, 'end' will be None."""
- def isFinished():
- """Return a boolean. True means the build has finished, False means
- it is still running."""
-
- def waitUntilFinished():
- """Return a Deferred that will fire when the build finishes. If the
- build has already finished, this deferred will fire right away. The
- callback is given this IBuildStatus instance as an argument."""
-
# while the build is running, the following methods make sense.
# Afterwards they return None
@@ -569,11 +642,9 @@
@rtype: implementor of L{IStatusReceiver}
"""
- def builderChangedState(builderName, state, eta=None):
+ def builderChangedState(builderName, state):
"""Builder 'builderName' has changed state. The possible values for
- 'state' are 'offline', 'idle', 'waiting', 'interlocked', and
- 'building'. For waiting and building, 'eta' gives the number of
- seconds from now that the state is expected to change."""
+ 'state' are 'offline', 'idle', and 'building'."""
def buildStarted(builderName, build):
"""Builder 'builderName' has just started a build. The build is an
@@ -652,12 +723,19 @@
themselves whether the change is interesting or not, and may initiate
a build as a result."""
+ def submitBuildSet(buildset):
+ """Submit a BuildSet object, which will eventually be run on all of
+ the builders listed therein."""
+ # TODO: return a status object
+
def getBuilder(name):
"""Retrieve the IBuilderControl object for the given Builder."""
class IBuilderControl(Interface):
- def forceBuild(who, reason): # TODO: add sourceStamp, patch
- """Start a build of the latest sources. If 'who' is not None, it is
+ def forceBuild(who, reason):
+ """DEPRECATED, please use L{requestBuild} instead.
+
+ Start a build of the latest sources. If 'who' is not None, it is
string with the name of the user who is responsible for starting the
build: they will be added to the 'interested users' list (so they may
be notified via email or another Status object when it finishes).
@@ -667,12 +745,22 @@
even if the Status object would normally only send results upon
failures.
- forceBuild() may raise NoSlaveError or BuilderInUseError if it
+ forceBuild() may raise L{NoSlaveError} or L{BuilderInUseError} if it
cannot start the build.
- forceBuild() returns an IBuildControl object which can be used to
- further control the new build, or from which an IBuildStatus object
- can be obtained."""
+ forceBuild() returns a Deferred which fires with an L{IBuildControl}
+ object that can be used to further control the new build, or from
+ which an L{IBuildStatus} object can be obtained."""
+
+ def requestBuild(request):
+ """Queue a L{buildbot.process.base.BuildRequest} object for later
+ building."""
+
+ def getPendingBuilds():
+ """Return a list of L{IBuildRequestControl} objects for this Builder.
+ Each one corresponds to a pending build that has not yet started (due
+ to a scarcity of build slaves). These upcoming builds can be canceled
+ through the control object."""
def getBuild(number):
"""Attempt to return an IBuildControl object for the given build.
@@ -690,10 +778,14 @@
# or something. However the event that is emitted is most useful in
# the Builder column, so it kinda fits here too.
+class IBuildRequestControl(Interface):
+ def cancel():
+ """Remove the build from the pending queue. Has no effect if the
+ build has already been started."""
+
class IBuildControl(Interface):
def getStatus():
"""Return an IBuildStatus object for the Build that I control."""
def stopBuild(reason="<no reason given>"):
"""Halt the build. This has no effect if the build has already
finished."""
-
--- NEW FILE: locks.py ---
# -*- test-case-name: buildbot.test.test_locks -*-
from twisted.python import log
from twisted.internet import reactor, defer
from buildbot import util
class BaseLock:
owner = None
description = "<BaseLock>"
def __init__(self, name):
self.name = name
self.waiting = []
def __repr__(self):
return self.description
def isAvailable(self):
log.msg("%s isAvailable: self.owner=%s" % (self, self.owner))
return not self.owner
def claim(self, owner):
log.msg("%s claim(%s)" % (self, owner))
assert owner is not None
self.owner = owner
def release(self, owner):
log.msg("%s release(%s)" % (self, owner))
assert owner is self.owner
self.owner = None
reactor.callLater(0, self.nowAvailable)
def waitUntilAvailable(self, owner):
log.msg("%s waitUntilAvailable(%s)" % (self, owner))
assert self.owner, "You aren't supposed to call this on a free Lock"
d = defer.Deferred()
self.waiting.append((d, owner))
return d
def nowAvailable(self):
log.msg("%s nowAvailable" % self)
assert not self.owner
if not self.waiting:
return
d,owner = self.waiting.pop(0)
d.callback(self)
class MasterLock(BaseLock, util.ComparableMixin):
compare_attrs = ['name']
def __init__(self, name):
BaseLock.__init__(self, name)
self.description = "<MasterLock(%s)>" % (name,)
def getLock(self, slave):
return self
class SlaveLock(util.ComparableMixin):
compare_attrs = ['name']
def __init__(self, name):
self.name = name
self.locks = {}
def getLock(self, slavebuilder):
slavename = slavebuilder.slave.slavename
if not self.locks.has_key(slavename):
lock = self.locks[slavename] = BaseLock(self.name)
lock.description = "<SlaveLock(%s)[%s]>" % (self.name, slavename)
self.locks[slavename] = lock
return self.locks[slavename]
Index: master.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/master.py,v
retrieving revision 1.73
retrieving revision 1.74
diff -u -d -r1.73 -r1.74
--- master.py 22 May 2005 02:16:14 -0000 1.73
+++ master.py 19 Jul 2005 23:11:59 -0000 1.74
@@ -71,7 +71,7 @@
"""
self.builders.remove(builder)
if self.slave:
- builder.detached()
+ builder.detached(self)
return self.sendBuilderList()
return defer.succeed(None)
@@ -238,7 +238,7 @@
# if we sent the builders list because of a config
# change, the Builder might already be attached.
# Builder.attached will ignore us if this happens.
- d = b.attached(remote, self.slave_commands)
+ d = b.attached(self, remote, self.slave_commands)
dl.append(d)
continue
return defer.DeferredList(dl)
@@ -270,7 +270,7 @@
self.slave = None
self.slave_status.connected = False
for b in self.builders:
- b.detached()
+ b.detached(self)
log.msg("Botmaster.detached(%s)" % self.slavename)
@@ -279,9 +279,6 @@
"""This is the master-side service which manages remote buildbot slaves.
It provides them with BotPerspectives, and distributes file change
notification messages to them.
-
- Any CVS changes that arrive should be handed to the .addChange method.
-
"""
debug = 0
@@ -300,20 +297,33 @@
def waitUntilBuilderAttached(self, name):
# convenience function for testing
- d = defer.Deferred()
b = self.builders[name]
+ if b.slaves:
+ return defer.succeed(None)
+ d = defer.Deferred()
b.watchers['attach'].append(d)
return d
def waitUntilBuilderDetached(self, name):
# convenience function for testing
- d = defer.Deferred()
- b = self.builders.get(name, None)
- if not b or not b.remote:
+ b = self.builders.get(name)
+ if not b or not b.slaves:
return defer.succeed(None)
+ d = defer.Deferred()
b.watchers['detach'].append(d)
return d
+ def waitUntilBuilderIdle(self, name):
+ # convenience function for testing
+ b = self.builders[name]
+ for sb in b.slaves.keys():
+ if b.slaves[sb] != "idle":
+ d = defer.Deferred()
+ b.watchers['idle'].append(d)
+ return d
+ return defer.succeed(None)
+
+
def addSlave(self, slavename):
slave = BotPerspective(slavename)
self.slaves[slavename] = slave
@@ -349,7 +359,7 @@
self.builders[builder.name] = builder
self.builderNames.append(builder.name)
builder.setBotmaster(self)
- self.checkInactiveInterlocks() # TODO?: do this in caller instead?
+ #self.checkInactiveInterlocks() # TODO?: do this in caller instead?
slave = self.slaves[slavename]
return slave.addBuilder(builder)
@@ -366,16 +376,16 @@
b = self.builders[builder.name]
# any linked interlocks will be made inactive before the builder is
# removed
- interlocks = []
- for i in b.feeders:
- assert i not in interlocks
- interlocks.append(i)
- for i in b.interlocks:
- assert i not in interlocks
- interlocks.append(i)
- for i in interlocks:
- if self.debug: print " deactivating interlock", i
- i.deactivate(self.builders)
+## interlocks = []
+## for i in b.feeders:
+## assert i not in interlocks
+## interlocks.append(i)
+## for i in b.interlocks:
+## assert i not in interlocks
+## interlocks.append(i)
+## for i in interlocks:
+## if self.debug: print " deactivating interlock", i
+## i.deactivate(self.builders)
del self.builders[builder.name]
self.builderNames.remove(builder.name)
slave = self.slaves.get(builder.slavename)
@@ -418,22 +428,6 @@
def getPerspective(self, slavename):
return self.slaves[slavename]
- def addChange(self, change):
- for b in self.builders.values():
- b.filesChanged(change)
-
- def forceBuild(self, name, reason="forced", who=None):
- """Manually tell a builder with the given name to start a build.
- Returns an IBuildControl object, which can be used to control or
- observe the build."""
-
- log.msg("BotMaster.forceBuild(%s)" % name)
- b = self.builders.get(name)
- if b:
- return b.forceBuild(who, reason)
- else:
- log.msg("unknown builder '%s'" % name)
-
def shutdownSlaves(self):
# TODO: make this into a bot method rather than a builder method
for b in self.slaves.values():
@@ -612,6 +606,7 @@
self.statusTargets = []
+ self.schedulers = []
self.bots = []
# this ChangeMaster is a dummy, only used by tests. In the real
# buildmaster, where the BuildMaster instance is activated
@@ -659,7 +654,6 @@
self.change_svc.disownServiceParent()
self.change_svc = changes
self.change_svc.basedir = self.basedir
- self.change_svc.botmaster = self.botmaster
self.change_svc.setName("changemaster")
self.dispatcher.changemaster = self.change_svc
self.change_svc.setServiceParent(self)
@@ -729,9 +723,9 @@
log.err("config file must define BuildmasterConfig")
raise
- known_keys = "bots sources builders slavePortnum " + \
+ known_keys = "bots sources schedulers builders slavePortnum " + \
"debugPassword manhole " + \
- "interlocks status projectName projectURL buildbotURL"
+ "status projectName projectURL buildbotURL"
known_keys = known_keys.split()
for k in config.keys():
if k not in known_keys:
@@ -741,13 +735,13 @@
# required
bots = config['bots']
sources = config['sources']
+ schedulers = config['schedulers']
builders = config['builders']
slavePortnum = config['slavePortnum']
# optional
debugPassword = config.get('debugPassword')
manhole = config.get('manhole')
- interlocks = config.get('interlocks', [])
status = config.get('status', [])
projectName = config.get('projectName')
projectURL = config.get('projectURL')
@@ -762,18 +756,21 @@
for name, passwd in bots:
if name in ("debug", "change", "status"):
raise KeyError, "reserved name '%s' used for a bot" % name
- for i in interlocks:
- name, feeders, watchers = i
- if type(feeders) != type([]):
- raise TypeError, "interlock feeders must be a list"
- if type(watchers) != type([]):
- raise TypeError, "interlock watchers must be a list"
- bnames = feeders + watchers
- for bname in bnames:
- if bnames.count(bname) > 1:
- why = ("builder '%s' appears multiple times for " + \
- "interlock %s") % (bname, name)
- raise ValueError, why
+ if config.has_key('interlocks'):
+ raise KeyError("c['interlocks'] is no longer accepted")
+
+## for i in interlocks:
+## name, feeders, watchers = i
+## if type(feeders) != type([]):
+## raise TypeError, "interlock feeders must be a list"
+## if type(watchers) != type([]):
+## raise TypeError, "interlock watchers must be a list"
+## bnames = feeders + watchers
+## for bname in bnames:
+## if bnames.count(bname) > 1:
+## why = ("builder '%s' appears multiple times for " + \
+## "interlock %s") % (bname, name)
+## raise ValueError, why
for s in status:
assert interfaces.IStatusReceiver(s)
@@ -796,6 +793,31 @@
% (b['name'], b['builddir']))
dirnames.append(b['builddir'])
+ # assert that all locks used by the Builds and their Steps are
+ # uniquely named.
+ locks = {}
+ for b in builders:
+ for l in b.get('locks', []):
+ if locks.has_key(l.name):
+ if locks[l.name] is not l:
+ raise ValueError("Two different locks (%s and %s) "
+ "share the name %s"
+ % (l, locks[l.name], l.name))
+ else:
+ locks[l.name] = l
+ # TODO: this will break with any BuildFactory that doesn't use a
+ # .steps list, but I think the verification step is more
+ # important.
+ for s in b['factory'].steps:
+ for l in s[1].get('locks', []):
+ if locks.has_key(l.name):
+ if locks[l.name] is not l:
+ raise ValueError("Two different locks (%s and %s)"
+ " share the name %s"
+ % (l, locks[l.name], l.name))
+ else:
+ locks[l.name] = l
+
# now we're committed to implementing the new configuration, so do
# it atomically
@@ -829,6 +851,7 @@
manhole.setServiceParent(self)
dl.append(self.loadConfig_Sources(sources))
+ dl.append(self.loadConfig_Schedulers(schedulers))
# add/remove self.botmaster.builders to match builders. The
# botmaster will handle startup/shutdown issues.
@@ -851,7 +874,7 @@
self.slavePortnum = slavePortnum
# self.interlocks:
- self.loadConfig_Interlocks(interlocks)
+ #self.loadConfig_Interlocks(interlocks)
log.msg("configuration updated")
self.readConfig = True
@@ -888,6 +911,15 @@
for source in sources if source not in self.change_svc]
return defer.DeferredList(dl)
+ def loadConfig_Schedulers(self, newschedulers):
+ old = [s for s in self.schedulers if s not in newschedulers]
+ [self.schedulers.remove(s) for s in old]
+ dl = [s.disownServiceParent() for s in old]
+ [s.setServiceParent(self)
+ for s in newschedulers if s not in self.schedulers]
+ self.schedulers = newschedulers
+ return defer.DeferredList(dl)
+
def loadConfig_Builders(self, newBuilders):
dl = []
old = self.botmaster.getBuildernames()
@@ -1001,6 +1033,27 @@
self.botmaster.addInterlock(i)
+ def addChange(self, change):
+ for s in self.schedulers:
+ s.addChange(change)
+
+ def submitBuildSet(self, bs):
+ # determine the set of Builders to use
+ builders = []
+ for name in bs.builderNames:
+ b = self.botmaster.builders.get(name)
+ if b:
+ if b not in builders:
+ builders.append(b)
+ continue
+ # TODO: add aliases like 'all'
+ raise KeyError("no such builder named '%s'" % name)
+
+ # now tell the BuildSet to create BuildRequests for all those
+ # Builders and submit them
+ bs.start(builders)
+
+
class Control:
if implements:
implements(interfaces.IControl)
@@ -1013,6 +1066,10 @@
def addChange(self, change):
self.master.change_svc.addChange(change)
+ def submitBuildSet(self, bs):
+ self.master.submitBuildSet(bs)
+ # TODO: return a BuildSetStatus
+
def getBuilder(self, name):
b = self.master.botmaster.builders[name]
return interfaces.IBuilderControl(b)
--- NEW FILE: scheduler.py ---
# -*- test-case-name: buildbot.test.test_dependencies -*-
import time
from twisted.internet import reactor
from twisted.application import service, internet
from twisted.python import log
from buildbot import interfaces, buildset, util
from buildbot.util import now
from buildbot.status import builder
from buildbot.twcompat import implements, providedBy
from buildbot.sourcestamp import SourceStamp
class BaseScheduler(service.MultiService, util.ComparableMixin):
if implements:
implements(interfaces.IScheduler)
else:
__implements__ = interfaces.IScheduler,
def __init__(self, name):
service.MultiService.__init__(self)
self.name = name
def __repr__(self):
return "<Scheduler '%s'>" % self.name
def submit(self, bs):
self.parent.submitBuildSet(bs)
class BaseUpstreamScheduler(BaseScheduler):
if implements:
implements(interfaces.IUpstreamScheduler)
else:
__implements__ = interfaces.IUpstreamScheduler,
def __init__(self, name):
BaseScheduler.__init__(self, name)
self.successWatchers = []
def subscribeToSuccessfulBuilds(self, watcher):
self.successWatchers.append(watcher)
def unsubscribeToSuccessfulBuilds(self, watcher):
self.successWatchers.remove(watcher)
def submit(self, bs):
d = bs.waitUntilFinished()
d.addCallback(self.buildSetFinished)
self.parent.submitBuildSet(bs)
def buildSetFinished(self, bss):
if not self.running:
return
if bss.getResults() == builder.SUCCESS:
ss = bss.getSourceStamp()
for w in self.successWatchers:
w(ss)
class Scheduler(BaseUpstreamScheduler):
"""The default Scheduler class will run a build after some period of time
called the C{treeStableTimer}, on a given set of Builders. It only pays
attention to a single branch. You you can provide a C{fileIsImportant}
function which will evaluate each Change to decide whether or not it
should trigger a new build.
"""
compare_attrs = ('name', 'treeStableTimer', 'builderNames', 'branch',
'fileIsImportant')
def __init__(self, name, branch, treeStableTimer, builderNames,
fileIsImportant=None):
"""
@param name: the name of this Scheduler
@param branch: The branch name that the Scheduler should pay
attention to. Any Change that is not on this branch
will be ignored. It can be set to None to only pay
attention to the default branch.
@param treeStableTimer: the duration, in seconds, for which the tree
must remain unchanged before a build will be
triggered. This is intended to avoid builds
of partially-committed fixes.
@param builderNames: a list of Builder names. When this Scheduler
decides to start a set of builds, they will be
run on the Builders named by this list.
@param fileIsImportant: A callable which takes one argument (a Change
instance) and returns True if the change is
worth building, and False if it is not.
Unimportant Changes are accumulated until the
build is triggered by an important change.
The default value of None means that all
Changes are important.
"""
BaseUpstreamScheduler.__init__(self, name)
self.treeStableTimer = treeStableTimer
for b in builderNames:
assert type(b) is str
self.builderNames = builderNames
self.branch = branch
if fileIsImportant:
assert callable(fileIsImportant)
self.fileIsImportant = fileIsImportant
self.importantChanges = []
self.unimportantChanges = []
self.nextBuildTime = None
self.timer = None
def fileIsImportant(self, change):
# note that externally-provided fileIsImportant callables are
# functions, not methods, and will only receive one argument. Or you
# can override this method, in which case it will behave like a
# normal method.
return True
def addChange(self, change):
if change.branch != self.branch:
log.msg("%s ignoring off-branch %s" % (self, change))
return
if self.fileIsImportant(change):
self.addImportantChange(change)
else:
self.addUnimportantChange(change)
def addImportantChange(self, change):
log.msg("%s: change is important, adding %s" % (self, change))
self.importantChanges.append(change)
self.nextBuildTime = max(self.nextBuildTime,
change.when + self.treeStableTimer)
self.setTimer(self.nextBuildTime)
def addUnimportantChange(self, change):
log.msg("%s: change is not important, adding %s" % (self, change))
self.unimportantChanges.append(change)
def setTimer(self, when):
log.msg("%s: setting timer to %s" %
(self, time.strftime("%H:%M:%S", time.localtime(when))))
now = util.now()
if when < now:
when = now + 1
if self.timer:
self.timer.cancel()
self.timer = reactor.callLater(when - now, self.fireTimer)
def stopTimer(self):
if self.timer:
self.timer.cancel()
self.timer = None
def fireTimer(self):
# clear out our state
self.timer = None
self.nextBuildTime = None
changes = self.importantChanges + self.unimportantChanges
self.importantChanges = []
self.unimportantChanges = []
# create a BuildSet, submit it to the BuildMaster
bs = buildset.BuildSet(self.builderNames,
SourceStamp(changes=changes))
self.submit(bs)
def stopService(self):
self.stopTimer()
return service.MultiService.stopService(self)
class AnyBranchScheduler(BaseUpstreamScheduler):
"""This Scheduler will handle changes on a variety of branches. It will
accumulate Changes for each branch separately. It works by creating a
separate Scheduler for each new branch it sees."""
schedulerFactory = Scheduler
compare_attrs = ('name', 'branches', 'treeStableTimer', 'builderNames',
'fileIsImportant')
def __init__(self, name, branches, treeStableTimer, builderNames,
fileIsImportant=None):
"""
@param name: the name of this Scheduler
@param branches: The branch names that the Scheduler should pay
attention to. Any Change that is not on one of these
branches will be ignored. It can be set to None to
accept changes from any branch.
@param treeStableTimer: the duration, in seconds, for which the tree
must remain unchanged before a build will be
triggered. This is intended to avoid builds
of partially-committed fixes.
@param builderNames: a list of Builder names. When this Scheduler
decides to start a set of builds, they will be
run on the Builders named by this list.
@param fileIsImportant: A callable which takes one argument (a Change
instance) and returns True if the change is
worth building, and False if it is not.
Unimportant Changes are accumulated until the
build is triggered by an important change.
The default value of None means that all
Changes are important.
"""
BaseUpstreamScheduler.__init__(self, name)
self.treeStableTimer = treeStableTimer
for b in builderNames:
assert type(b) is str
self.builderNames = builderNames
self.branches = branches
if fileIsImportant:
assert callable(fileIsImportant)
self.fileIsImportant = fileIsImportant
self.schedulers = {} # one per branch
def addChange(self, change):
branch = change.branch
if self.branches and branch not in self.branches:
log.msg("%s ignoring off-branch %s" % (self, change))
return
s = self.schedulers.get(branch)
if not s:
name = self.name + "." + branch
s = self.schedulerFactory(name, branch,
self.treeStableTimer,
self.builderNames,
self.fileIsImportant)
s.successWatchers = self.successWatchers
s.setServiceParent(self)
# TODO: does this result in schedulers that stack up forever?
# When I make the persistify-pass, think about this some more.
self.schedulers[branch] = s
s.addChange(change)
def submitBuildSet(self, bs):
self.parent.submitBuildSet(bs)
class Dependent(BaseUpstreamScheduler):
"""This scheduler runs some set of 'downstream' builds when the
'upstream' scheduler has completed successfully."""
compare_attrs = ('name', 'upstream', 'builders')
def __init__(self, name, upstream, builderNames):
assert providedBy(upstream, interfaces.IUpstreamScheduler)
BaseUpstreamScheduler.__init__(self, name)
self.upstream = upstream
self.builderNames = builderNames
def startService(self):
service.MultiService.startService(self)
self.upstream.subscribeToSuccessfulBuilds(self.upstreamBuilt)
def stopService(self):
d = service.MultiService.stopService(self)
self.upstream.unsubscribeToSuccessfulBuilds(self.upstreamBuilt)
return d
def upstreamBuilt(self, ss):
bs = buildset.BuildSet(self.builderNames, ss)
self.submit(bs)
class Periodic(BaseUpstreamScheduler):
"""Instead of watching for Changes, this Scheduler can just start a build
at fixed intervals. The C{periodicBuildTimer} parameter sets the number
of seconds to wait between such periodic builds. The first build will be
run immediately."""
# TODO: consider having this watch another (changed-based) scheduler and
# merely enforce a minimum time between builds.
compare_attrs = ('name', 'builderNames', 'periodicBuildTimer', 'branch')
def __init__(self, name, builderNames, periodicBuildTimer,
branch=None):
BaseUpstreamScheduler.__init__(self, name)
self.builderNames = builderNames
self.periodicBuildTimer = periodicBuildTimer
self.branch = branch
self.timer = internet.TimerService(self.periodicBuildTimer,
self.doPeriodicBuild)
self.timer.setServiceParent(self)
def doPeriodicBuild(self):
bs = buildset.BuildSet(self.builderNames,
SourceStamp(branch=self.branch))
self.submit(bs)
--- NEW FILE: sourcestamp.py ---
from buildbot import util, interfaces
from buildbot.twcompat import implements
class SourceStamp(util.ComparableMixin):
"""
a tuple of (branch, revision, patchspec, changes).
C{branch} is always valid, although it may be None to let the Source
step use its default branch. There are four possibilities for the
remaining elements:
- (revision=REV, patchspec=None, changes=None): build REV
- (revision=REV, patchspec=(LEVEL, DIFF), changes=None): checkout REV,
then apply a patch to the source, with C{patch -pPATCHLEVEL <DIFF}.
- (revision=None, patchspec=None, changes=[CHANGES]): let the Source
step check out the latest revision indicated by the given Changes.
CHANGES is a list of L{buildbot.changes.changes.Change} instances,
and all must be on the same branch.
- (revision=None, patchspec=None, changes=None): build the latest code
from the given branch.
"""
# all four of these are publically visible attributes
branch = None
revision = None
patch = None
changes = []
compare_attrs = ('branch', 'revision', 'patch', 'changes')
if implements:
implements(interfaces.ISourceStamp)
else:
__implements__ = interfaces.ISourceStamp,
def __init__(self, branch=None, revision=None, patch=None,
changes=None):
self.branch = branch
self.revision = revision
self.patch = patch
if changes:
self.changes = changes
self.branch = changes[0].branch
def canBeMergedWith(self, other):
if other.branch != self.branch:
return False # the builds are completely unrelated
if self.changes and other.changes:
# TODO: consider not merging these. It's a tradeoff between
# minimizing the number of builds and obtaining finer-grained
# results.
return True
elif self.changes and not other.changes:
return False # we're using changes, they aren't
elif not self.changes and other.changes:
return False # they're using changes, we aren't
if self.patch or other.patch:
return False # you can't merge patched builds with anything
if self.revision == other.revision:
# both builds are using the same specific revision, so they can
# be merged. It might be the case that revision==None, so they're
# both building HEAD.
return True
return False
def mergeWith(self, others):
"""Generate a SourceStamp for the merger of me and all the other
BuildRequests. This is called by a Build when it starts, to figure
out what its sourceStamp should be."""
# either we're all building the same thing (changes==None), or we're
# all building changes (which can be merged)
changes = []
changes.extend(self.changes)
for req in others:
assert self.canBeMergedWith(req) # should have been checked already
changes.extend(req.changes)
newsource = SourceStamp(branch=self.branch,
revision=self.revision,
patch=self.patch,
changes=changes)
return newsource
- Previous message (by thread): [Buildbot-commits] buildbot/docs .arch-inventory,1.2,1.3 Makefile,1.1,1.2 buildbot.texinfo,1.8,1.9 .cvsignore,1.1,1.2
- Next message (by thread): [Buildbot-commits] buildbot ChangeLog,1.466,1.467 Makefile,1.13,1.14
- Messages sorted by:
[ date ]
[ thread ]
[ subject ]
[ author ]
More information about the Commits
mailing list