[Buildbot-devel] trying to use buildbot for freeware scientific data package...
Brian Warner
warner at lothar.com
Thu Jan 8 18:50:32 UTC 2004
From: Ed Hartnett <ed at unidata.ucar.edu>
> Thanks for your help, I am definitely making progress here...
>
> But my master still won't start up, not being able to find a cvs
> directory, even though it is there.
>
> I don't want to use CVSToys, because my CVS server is on another
> machine, and I don't want to mess with it. (Lot's of others use it as
> well).
>
> How do I set up email notification to go into my email directory?
There are two parts. The first is to set up your Mail Delivery Agent (MDA) to
deposit the right messages in a maildir. The second is to tell the
buildmaster to watch that maildir for messages.
> What is the meaning of the comment in your twisted master file:
>
> # use with: echo "$BASEDIR/cvsmail/" >~/.qmail
That's the MDA half. The mail system I use is called "qmail", and when it
delivers a message it will obey directives in a file called ~/.qmail or
something similar. If that file has a line that ends in a "/" and doesn't
start with a "|", it will treat that line as a maildir which the message
should be written into.
maildirs are simply a triplet of subdirectories like so:
my-maildir/tmp/
my-maildir/new/
my-maildir/cur/
And the "maildir protocol" is to write the file into the tmp/ directory,
fsync it, then move it into the new/ directory. Anything in the new/
directory is considered to be in the "inbox". The file gets a randomly-chosen
name, so it shouldn't overlap with anything already in new/ . By fully
writing the message out to disk before moving it, the act of adding it to the
inbox is atomic and cannot fail (due to lack of disk space, etc).
(when your MUA pulls the message out of the inbox, it can just move it from
new/ to cur/ . The buildmaster does this once it has successfully parsed
the message.)
There is a tool called "safecat", written by Len Budney, which will accept
text on stdin and write it into a maildir. Other MDAs know about maildirs: I
know you can set up a procmailrc to deliver into one.
http://www.pobox.com/~lbudney/linux/software/safecat.html
The nice thing about qmail (and also postfix if I recall correctly) is that
it is easy to set up extension addresses. Mail to 'warner at lothar.com' will
obey the commands in my ~warner/.qmail file. Mail to
'warner-buildbot at lothar.com' will obey the commands in my
~warner/.qmail-buildbot file. This means you can use your own address to feed
the buildmaster and have those messages be processed completely independently
from your normal mail. Other MDAs (like the one sendmail uses) will obey a
single .forward file, so you'd either need to set up a separate account just
for the buildmaster (which isn't a bad idea, really), or do some kind of
fancy parsing in your .forward to separate your messages into
buildmaster-related vs. personal mail, and then deposit them in the right
places.
> source = FCMaildirSource(os.path.join(basedir, "cvsmail"),
> prefix="Twisted")
> c['sources'].append(source)
That's the buildmaster half. What you're doing there is telling it that there
is a maildir under the base directory (which is the one you give to mktap
when you build the buildmaster .tap file) called "cvsmail/", and that the
buildmaster should watch for messages to appear there and then parse them
using the FreshCVS mail parser function.
If you created the buildmaster with --basedir=~/BuildBot/master , then this
source would watch ~/BuildBot/master/cvsmail/new/ for messages to appear. You
would then put "BuildBot/master/cvsmail/" in a .qmail file (such directories
are evaluated relative to your home directory), or use a .forward with
something like:
|safecat ~/BuildBot/master/cvsmail/tmp ~/BuildBot/master/cvsmail/new
(safecat takes the pair of directories as arguments instead of the single
common parent directory.. see the safecat docs for details)
> When I start my slave builder it works, but my master builder doesn't
> start up. :-(
>
> Here's the log output:
>
>
> 2004/01/08 10:19 MST [-] twistd 1.1.1 (/usr/bin/python 2.2.3) starting up
> 2004/01/08 10:19 MST [-] reactor class: twisted.internet.default.SelectReactor
> 2004/01/08 10:19 MST [-] Loading buildbot.tap...
> 2004/01/08 10:19 MST [-] Loaded.
> 2004/01/08 10:19 MST [-] set uid/gid 4178/2000
> 2004/01/08 10:19 MST [-] loading configuration from /home/ed/BuildBot/master.cfg
> 2004/01/08 10:19 MST [-] unknown key 'basedir' defined in config dictionary
> 2004/01/08 10:19 MST [-] error during loadConfig
> 2004/01/08 10:19 MST [-] Traceback (most recent call last):
> File "/usr/lib/python2.2/site-packages/twisted/scripts/twistd.py", line 165, in startApplication
> app.startApplication(application, not config['no_save'])
> File "/usr/lib/python2.2/site-packages/twisted/application/app.py", line 220, in startApplication
> service.IService(application).startService()
> File "/usr/lib/python2.2/site-packages/twisted/application/service.py", line 194, in startService
> service.startService()
> File "/usr/lib/python2.2/site-packages/buildbot/master.py", line 378, in startService
> self.loadTheConfigFile()
> --- <exception caught here> ---
> File "/usr/lib/python2.2/site-packages/buildbot/master.py", line 397, in loadTheConfigFile
> self.loadConfig(f)
> File "/usr/lib/python2.2/site-packages/buildbot/master.py", line 485, in loadConfig
> for source in new if source not in old]
> File "/usr/lib/python2.2/site-packages/buildbot/changes/changes.py", line 110, in addSource
> source.setServiceParent(self)
> File "/usr/lib/python2.2/site-packages/twisted/application/service.py", line 116, in setServiceParent
> self.parent.addService(self)
> File "/usr/lib/python2.2/site-packages/twisted/application/service.py", line 220, in addService
> service.startService()
> File "/usr/lib/python2.2/site-packages/buildbot/changes/maildirtwisted.py", line 21, in startService
> self.start()
> File "/usr/lib/python2.2/site-packages/buildbot/changes/maildir.py", line 43, in start
> raise "invalid maildir '%s'" % self.basedir
> invalid maildir '/home/ed/BuildBot/cvsmail': None
>
>
> Here's my master.cfg file:
>
> #! /usr/bin/python
>
> # You need this next line if using CVSToys
> #from buildbot.changes.freshcvs import FreshCVSSource
>
> from buildbot.changes.freshcvsmail import FCMaildirSource
> from buildbot.process.base import \
> BasicBuildFactory, QuickBuildFactory
> import os.path
>
> c = {}
>
> # Set up one bot, on Ed's Linux machine.
> c['bots'] = [["EdsLinuxBox1", "sekrit"]]
>
> # Specify the CVS host info: host, port, user, passwd, prefix. Use
> # this if you have CVSToys.
>
> #c['sources'] = [FreshCVSSource("localhost", 4519,
> # "ed", "edspassword",
> # prefix="netcdf-4")]
>
> # Use this for CVS email notification.
> source = FCMaildirSource(os.path.join(basedir, "cvsmail"),
> prefix="netcdf-3")
> c['sources'] = []
> c['sources'].append(source)
>
>
> c['builders'] = []
>
> repository = "/upc/share/CVS"
> cvsmodule = "netcdf-3"
> # Parameters: cvsroot, cvsmodule, workdir="build", configure=None,
> # configureEnv={}, compile="make all", test="make check", cvsCopy=0
> f1 = QuickBuildFactory(repository, cvsmodule,
> #configure="./configure --disable-shared",
> configureEnv={'FC': '', 'CXX':'', 'F90':''},
> # configure=None
> )
> c['builders'].append(("netcdf-3-c-only", "EdsLinuxBox1", "netcdf-3-c-only", f1))
>
> #c['irc'] = {("localhost", 6667): ('buildbot', ["private"])}
>
> c['slavePortnum'] = 8007
> c['webPortnum'] = 8080
> c['debugPassword'] = "asdf"
>
> BuildmasterConfig = c
The first thing to be aware of is that "basedir" was not inserted into the
config file's namespace until buildbot-0.4.1 (earlier versions had examples
which suggested it was available, but it was not). That wouldn't cause the
problem your log shows, but it's a good thing to keep in mind.
> 2004/01/08 10:19 MST [-] unknown key 'basedir' defined in config dictionary
Something looks fishy here. This line suggests that your master.cfg file has
a line in it like:
c['basedir'] = "foo"
Which wouldn't be right (and isn't in the file you attached).. the basedir is
set when you do mktap, and is an *input* to the master.cfg file (i.e. it is
available but read-only within the config file, and cannot be changed from
there [since that would then change the directory to look for the config
file]). If you would, double-check that your master.cfg file isn't doing
something like this, and then re-start the buildmaster and see if that
message is still present. If so, let me know.. maybe there's a bug.
It might also be possible that you're using a .tap file generated under and
earlier version of the buildbot code. This may cause confusion (it's a bug,
but not one that's particularly easy to fix so it's lower down on the
priority list). If that's a possibility, re-run mktap. In general you want to
run mktap and the buildmaster itself (under twistd) with the same versions of
the buildbot code installed.
> File "/usr/lib/python2.2/site-packages/buildbot/changes/maildir.py", line 43, in start
> raise "invalid maildir '%s'" % self.basedir
> invalid maildir '/home/ed/BuildBot/cvsmail': None
That exception means that the FCMaildirSource wasn't able to read a directory
named "/home/ed/BuildBot/cvsmail/new". It could be that the directory doesn't
exist, or that the permissions aren't set up to allow the user running the
buildmaster read from it.
If you are running buildbot-0.4.0, you can just manually insert the base
directory to work with.. something like:
source = FCMaildirSource("/home/ed/BuildBot/cvsmail"),
prefix="netcdf-3")
the os.path.join is just a convenience: it lets you relocate the entire
buildmaster by rebuilding the .tap file with a different --basedir argument,
without having to modify the master.cfg file as well.
Let me know if that helps or if you run into problems..
-Brian
More information about the devel
mailing list