[Buildbot-devel] monotone and buildbot
Nathaniel Smith
njs at pobox.com
Sun Mar 27 00:57:27 UTC 2005
I'm trying to set up a buildbot for monotone: http://venge.net/monotone/
Monotone is, of course, developed using monotone, which makes this a
somewhat more complex project than it might otherwise be :-). I've
been trying to add monotone support to buildbot, and would appreciate
some feedback.
The basic challenge is that monotone makes some quite different
assumptions from other revision control tools. In particular, it has
no concept of a branch that grows linearly over time, one change
after another. Instead, each checkin is represented by a SHA1
describing its contents, and a branch is a DAG of these checkins. So
we have the concept of a branch "head", but -- there may be multiple
heads. Obviously, this makes things for buildbot a little confusing.
We can't just accumulate up changes until the tree is steady and then
build the last of them, because there is no "last one". Force builds
also seem to work wierd, since as I understand they're expected to
just build "current tip", and again, there's no such thing.
I think the best behavior would be, whenever a builder goes idle,
check to see if there are any new heads that haven't been built yet.
If so, pick one randomly, and build it. (Or you could play around
wiht the heuristics for picking one, I suppose.) Keep the linear
display and everything, that works out okay, since development is
overall generally linear and there are rarely more than 2-3 parallel
development lines, so just mushing them into a single linear timeline
for display works.
It would sort of work for each builder to look at all newly arrived
revisions, pick a head one, and build it; but then there would need to
be some way to inform the changemaster that "actually, I only built
half of those changes you gave me, so give me the other half again
when I'm done", or something?
It also seems that forced builds need some way to specify which
revision to build. "current head" is neither well-defined, nor
actually very useful; usually I want to know whether a particular tree
I just committed broke anything on platform X, not whether some
randomly chosen recent tree broke anything on platform X :-).
Any thoughts on how to go about these things? What part of
buildbot's source code should I be looking at? I don't understand the
logic for generating and using changes very well yet; it seems that in
my current first-pass implementation, doing a "force build" passes a
null changes vector to my builder; but for some reason, an actual
change (as seen in the master logs and on the web page) doesn't cause
any builders to trigger at all. Maybe because for now I'm just
passing an empty files list :-)
In the future, it would also be nice to take actions based on the
results of builds -- for instance, monotone has the capability to mark
in the repository whether a given revision works or not, and then make
use of this information at 'update' time. We also do various sorts of
metrics tracking -- currently test coverage, hopefully performance
benchmarks soon as well -- and it would be nice to tie these into the
buildbot change trigger stuff instead of using the current ad hoc mess
of shell scripts. Where would I look to hook things like this in?
Thanks a lot,
-- Nathaniel
--
"...these, like all words, have single, decontextualized meanings: everyone
knows what each of these words means, everyone knows what constitutes an
instance of each of their referents. Language is fixed. Meaning is
certain. Santa Claus comes down the chimney at midnight on December 24."
-- The Language War, Robin Lakoff
-------------- next part --------------
? buildbot-monotone-first-pass.diff
? buildbot-test-vc-1
? buildbot-test-vc-1.tar.gz
? m
? s
? buildbot/changes/monotone.py
Index: buildbot/master.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/master.py,v
retrieving revision 1.57
diff -u -r1.57 master.py
--- buildbot/master.py 6 Dec 2004 07:36:34 -0000 1.57
+++ buildbot/master.py 27 Mar 2005 00:58:53 -0000
@@ -657,16 +657,15 @@
f.close()
def loadConfig(self, f):
- globalDict = {}
- localDict = {'basedir': os.path.expanduser(self.basedir)}
+ env = {"basedir": os.path.expanduser(self.basedir)}
try:
- exec f in globalDict, localDict
+ exec f in env
except:
log.msg("error while parsing config file")
raise
try:
- config = localDict['BuildmasterConfig']
+ config = env['BuildmasterConfig']
except KeyError:
log.err("missing config dictionary")
log.err("config file must define BuildmasterConfig")
Index: buildbot/process/step.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/process/step.py,v
retrieving revision 1.59
diff -u -r1.59 step.py
--- buildbot/process/step.py 9 Dec 2004 10:24:56 -0000 1.59
+++ buildbot/process/step.py 27 Mar 2005 00:58:53 -0000
@@ -1088,6 +1088,38 @@
self.cmd = LoggedRemoteCommand("arch", self.args)
ShellCommand.start(self)
+class Monotone(Source):
+ """Check out a revision from a monotone server at 'server_addr',
+ collection 'collection'. 'revision' specifies which revision id to check
+ out.
+
+ This step will first create a local database, if necessary, and then pull
+ the contents of the server into the database. Then it will do the
+ checkout/update from this database."""
+
+ name = "monotone"
+
+ def __init__(self, server_addr, collection, db_path="monotone.db",
+ monotone="monotone",
+ **kwargs):
+ Source.__init__(self, **kwargs)
+ self.args.update({"server_addr": server_addr,
+ "collection": collection,
+ "db_path": db_path,
+ "monotone": monotone})
+
+ def computeSourceRevision(self, changes):
+ if not changes:
+ return None
+ return changes[-1].revision
+
+ def startVC(self):
+ slavever = self.slaveVersion("monotone")
+ assert slavever, "slave is too old, does not know about monotone"
+ self.cmd = LoggedRemoteCommand("monotone", self.args)
+ ShellCommand.start(self)
+
+
class todo_P4(Source):
name = "p4"
Index: buildbot/slave/commands.py
===================================================================
RCS file: /cvsroot/buildbot/buildbot/buildbot/slave/commands.py,v
retrieving revision 1.19
diff -u -r1.19 commands.py
--- buildbot/slave/commands.py 6 Dec 2004 09:00:32 -0000 1.19
+++ buildbot/slave/commands.py 27 Mar 2005 00:58:54 -0000
@@ -798,6 +798,97 @@
registerSlaveCommand("darcs", Darcs, cvs_ver)
+class Monotone(SourceBase):
+ """Monotone-specific VC operation. In addition to the arguments handled
+ by SourceBase, this command reads the following keys:
+
+ ['server_addr'] (required): the address of the server to pull from
+ ['collection'] (required): the name of the collection to pull
+ ['db_path'] (required): the local database path to use
+ ['revision'] (required): the revision to check out
+ ['monotone']: (required): path to monotone executable
+ """
+
+ header = "monotone operation"
+
+ def setup(self, args):
+ SourceBase.setup(self, args)
+ self.server_addr = args["server_addr"]
+ self.collection = args["collection"]
+ self.db_path = args["db_path"]
+ self.revision = args["revision"]
+ self.monotone = args["monotone"]
+ self._made_fulls = False
+ self._pull_timeout = args["timeout"]
+
+ def _makefulls(self):
+ if not self._made_fulls:
+ self.full_db_path = os.path.join(self.builder.basedir, self.db_path)
+ self.full_srcdir = os.path.join(self.builder.basedir, self.srcdir)
+ self._made_fulls = True
+
+ def sourcedirIsUpdateable(self):
+ self._makefulls()
+ if os.path.exists(os.path.join(self.full_srcdir,
+ ".buildbot_patched")):
+ return False
+ return (os.path.isfile(self.full_db_path)
+ and os.path.isdir(os.path.join(self.full_srcdir, "MT")))
+
+ def doVCUpdate(self):
+ self._makefulls()
+ # update: possible for mode in ('copy', 'update')
+ command = [self.monotone, "update", self.revision]
+ c = ShellCommand(self.builder, command, self.full_srcdir,
+ sendRC=False, timeout=self.timeout)
+ self.command = c
+ return c.start()
+
+ def doVCFull(self):
+ self._makefulls()
+ # to do a checkout, we first have to make sure the database exists,
+ # then pull into it.
+ if os.path.isfile(self.full_db_path):
+ command = ["true"]
+ else:
+ # We'll be doing an initial pull, so up the timeout to 3 hours so
+ # it will have time to complete.
+ self._pull_timeout = max(self._pull_timeout, 3 * 60 * 60)
+ self.sendStatus({"header": "creating database %s\n"
+ % (self.full_db_path,)})
+ command = [self.monotone, "db", "init", "--db=" + self.full_db_path]
+ c = ShellCommand(self.builder, command, self.builder.basedir,
+ sendRC=False, timeout=self.timeout)
+ self.command = c
+ d = c.start()
+ d.addCallback(self._abandonOnFailure)
+ d.addCallback(self._didDbInit)
+ d.addCallback(self._didPull)
+ return d
+
+ def _didDbInit(self, res):
+ # We have a database; let's pull into it.
+ command = [self.monotone, "--db=" + self.full_db_path,
+ "pull", "--ticker=dot", self.server_addr, self.collection]
+ c = ShellCommand(self.builder, command, self.builder.basedir,
+ sendRC=False, timeout=self.timeout)
+ self.sendStatus({"header": "pulling %s from %s\n"
+ % (self.collection, self.server_addr)})
+ self.command = c
+ return c.start()
+
+ def _didPull(self, res):
+ # We have a pulled database. Time to checkout!
+ command = [self.monotone, "--db=" + self.full_db_path,
+ "checkout", self.revision, self.full_srcdir]
+ c = ShellCommand(self.builder, command, self.builder.basedir,
+ sendRC=False, timeout=self.timeout)
+ self.command = c
+ return c.start()
+
+registerSlaveCommand("monotone", Monotone, cvs_ver)
+
+
class Arch(SourceBase):
"""Arch-specific (tla-specific) VC operation. In addition to the
arguments handled by SourceBase, this command reads the following keys:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: changes_monotone.py
Type: text/x-python
Size: 6800 bytes
Desc: not available
URL: <http://buildbot.net/pipermail/devel/attachments/20050326/8102cc36/attachment.py>
More information about the devel
mailing list