[Buildbot-devel] trying to use buildbot for freeware scientific data package...

[Ed Hartnett]

Brian Warner <warner at lothar.com> writes:

> From: Ed Hartnett <ed at unidata.ucar.edu>
> 
> > Thanks for your help, I am definitely making progress here...
> > 
> > But my master still won't start up, not being able to find a cvs
> > directory, even though it is there.
> > 
> > I don't want to use CVSToys, because my CVS server is on another
> > machine, and I don't want to mess with it. (Lot's of others use it as
> > well).
> > 
> > How do I set up email notification to go into my email directory?
> 
> There are two parts. The first is to set up your Mail Delivery Agent (MDA) to
> deposit the right messages in a maildir. The second is to tell the
> buildmaster to watch that maildir for messages.
> 
> > What is the meaning of the comment in your twisted master file:
> > 
> >     # use with:  echo "$BASEDIR/cvsmail/" >~/.qmail
> 
> That's the MDA half. The mail system I use is called "qmail", and when it
> delivers a message it will obey directives in a file called ~/.qmail or
> something similar. If that file has a line that ends in a "/" and doesn't
> start with a "|", it will treat that line as a maildir which the message
> should be written into.
> 
> maildirs are simply a triplet of subdirectories like so:
> 
>  my-maildir/tmp/
>  my-maildir/new/
>  my-maildir/cur/
> 
> And the "maildir protocol" is to write the file into the tmp/ directory,
> fsync it, then move it into the new/ directory. Anything in the new/
> directory is considered to be in the "inbox". The file gets a randomly-chosen
> name, so it shouldn't overlap with anything already in new/ . By fully
> writing the message out to disk before moving it, the act of adding it to the
> inbox is atomic and cannot fail (due to lack of disk space, etc).
> 
>  (when your MUA pulls the message out of the inbox, it can just move it from
>   new/ to cur/ . The buildmaster does this once it has successfully parsed
>   the message.)
> 
> There is a tool called "safecat", written by Len Budney, which will accept
> text on stdin and write it into a maildir. Other MDAs know about maildirs: I
> know you can set up a procmailrc to deliver into one.
> 
>  http://www.pobox.com/~lbudney/linux/software/safecat.html
> 
> The nice thing about qmail (and also postfix if I recall correctly) is that
> it is easy to set up extension addresses. Mail to 'warner at lothar.com' will
> obey the commands in my ~warner/.qmail file. Mail to
> 'warner-buildbot at lothar.com' will obey the commands in my
> ~warner/.qmail-buildbot file. This means you can use your own address to feed
> the buildmaster and have those messages be processed completely independently
> >From your normal mail. Other MDAs (like the one sendmail uses) will obey a
> single .forward file, so you'd either need to set up a separate account just
> for the buildmaster (which isn't a bad idea, really), or do some kind of
> fancy parsing in your .forward to separate your messages into
> buildmaster-related vs. personal mail, and then deposit them in the right
> places.
> 
> >     source = FCMaildirSource(os.path.join(basedir, "cvsmail"),
> >                              prefix="Twisted")
> >     c['sources'].append(source)
> 
> That's the buildmaster half. What you're doing there is telling it that there
> is a maildir under the base directory (which is the one you give to mktap
> when you build the buildmaster .tap file) called "cvsmail/", and that the
> buildmaster should watch for messages to appear there and then parse them
> using the FreshCVS mail parser function.
> 
> If you created the buildmaster with --basedir=~/BuildBot/master , then this
> source would watch ~/BuildBot/master/cvsmail/new/ for messages to appear. You
> would then put "BuildBot/master/cvsmail/" in a .qmail file (such directories
> are evaluated relative to your home directory), or use a .forward with
> something like:
> 
>  |safecat ~/BuildBot/master/cvsmail/tmp ~/BuildBot/master/cvsmail/new
> 
> (safecat takes the pair of directories as arguments instead of the single
> common parent directory.. see the safecat docs for details)
> 
> > When I start my slave builder it works, but my master builder doesn't
> > start up. :-(
> > 
> > Here's the log output:
> > 
> > 
> > 2004/01/08 10:19 MST [-] twistd 1.1.1 (/usr/bin/python 2.2.3) starting up
> > 2004/01/08 10:19 MST [-] reactor class: twisted.internet.default.SelectReactor
> > 2004/01/08 10:19 MST [-] Loading buildbot.tap...
> > 2004/01/08 10:19 MST [-] Loaded.
> > 2004/01/08 10:19 MST [-] set uid/gid 4178/2000
> > 2004/01/08 10:19 MST [-] loading configuration from /home/ed/BuildBot/master.cfg
> > 2004/01/08 10:19 MST [-] unknown key 'basedir' defined in config dictionary
> > 2004/01/08 10:19 MST [-] error during loadConfig
> > 2004/01/08 10:19 MST [-] Traceback (most recent call last):
> > 	  File "/usr/lib/python2.2/site-packages/twisted/scripts/twistd.py", line 165, in startApplication
> > 	    app.startApplication(application, not config['no_save'])
> > 	  File "/usr/lib/python2.2/site-packages/twisted/application/app.py", line 220, in startApplication
> > 	    service.IService(application).startService()
> > 	  File "/usr/lib/python2.2/site-packages/twisted/application/service.py", line 194, in startService
> > 	    service.startService()
> > 	  File "/usr/lib/python2.2/site-packages/buildbot/master.py", line 378, in startService
> > 	    self.loadTheConfigFile()
> > 	--- <exception caught here> ---
> > 	  File "/usr/lib/python2.2/site-packages/buildbot/master.py", line 397, in loadTheConfigFile
> > 	    self.loadConfig(f)
> > 	  File "/usr/lib/python2.2/site-packages/buildbot/master.py", line 485, in loadConfig
> > 	    for source in new if source not in old]
> > 	  File "/usr/lib/python2.2/site-packages/buildbot/changes/changes.py", line 110, in addSource
> > 	    source.setServiceParent(self)
> > 	  File "/usr/lib/python2.2/site-packages/twisted/application/service.py", line 116, in setServiceParent
> > 	    self.parent.addService(self)
> > 	  File "/usr/lib/python2.2/site-packages/twisted/application/service.py", line 220, in addService
> > 	    service.startService()
> > 	  File "/usr/lib/python2.2/site-packages/buildbot/changes/maildirtwisted.py", line 21, in startService
> > 	    self.start()
> > 	  File "/usr/lib/python2.2/site-packages/buildbot/changes/maildir.py", line 43, in start
> > 	    raise "invalid maildir '%s'" % self.basedir
> > 	invalid maildir '/home/ed/BuildBot/cvsmail': None
> > 	
> > 
> > Here's my master.cfg file:
> > 
> > #! /usr/bin/python
> > 
> > # You need this next line if using CVSToys
> > #from buildbot.changes.freshcvs import FreshCVSSource
> > 
> > from buildbot.changes.freshcvsmail import FCMaildirSource
> > from buildbot.process.base import \
> >      BasicBuildFactory, QuickBuildFactory
> > import os.path
> > 
> > c = {}
> > 
> > # Set up one bot, on Ed's Linux machine.
> > c['bots'] = [["EdsLinuxBox1", "sekrit"]]
> > 
> > # Specify the CVS host info: host, port, user, passwd, prefix. Use
> > # this if you have CVSToys.
> > 
> > #c['sources'] = [FreshCVSSource("localhost", 4519,
> > #                               "ed", "edspassword",
> > #                               prefix="netcdf-4")]
> > 
> > # Use this for CVS email notification.
> > source = FCMaildirSource(os.path.join(basedir, "cvsmail"),
> >                          prefix="netcdf-3")
> > c['sources'] = []
> > c['sources'].append(source)
> > 
> > 
> > c['builders'] = []
> > 
> > repository = "/upc/share/CVS"
> > cvsmodule = "netcdf-3"
> > # Parameters: cvsroot, cvsmodule, workdir="build", configure=None,
> > # configureEnv={}, compile="make all", test="make check", cvsCopy=0
> > f1 = QuickBuildFactory(repository, cvsmodule,
> >                        #configure="./configure --disable-shared",
> >                        configureEnv={'FC': '', 'CXX':'', 'F90':''},
> > #                       configure=None
> >                        )
> > c['builders'].append(("netcdf-3-c-only", "EdsLinuxBox1", "netcdf-3-c-only", f1))
> > 
> > #c['irc'] = {("localhost", 6667): ('buildbot', ["private"])}
> > 
> > c['slavePortnum'] = 8007
> > c['webPortnum'] = 8080
> > c['debugPassword'] = "asdf"
> > 
> > BuildmasterConfig = c
> 
> The first thing to be aware of is that "basedir" was not inserted into the
> config file's namespace until buildbot-0.4.1 (earlier versions had examples
> which suggested it was available, but it was not). That wouldn't cause the
> problem your log shows, but it's a good thing to keep in mind.
> 
> > 2004/01/08 10:19 MST [-] unknown key 'basedir' defined in config dictionary
> 
> Something looks fishy here. This line suggests that your master.cfg file has
> a line in it like:
> 
>  c['basedir'] = "foo"
> 
> Which wouldn't be right (and isn't in the file you attached).. the basedir is
> set when you do mktap, and is an *input* to the master.cfg file (i.e. it is
> available but read-only within the config file, and cannot be changed from
> there [since that would then change the directory to look for the config
> file]). If you would, double-check that your master.cfg file isn't doing
> something like this, and then re-start the buildmaster and see if that
> message is still present. If so, let me know.. maybe there's a bug.
> 
> It might also be possible that you're using a .tap file generated under and
> earlier version of the buildbot code. This may cause confusion (it's a bug,
> but not one that's particularly easy to fix so it's lower down on the
> priority list). If that's a possibility, re-run mktap. In general you want to
> run mktap and the buildmaster itself (under twistd) with the same versions of
> the buildbot code installed.
> 
> > 	  File "/usr/lib/python2.2/site-packages/buildbot/changes/maildir.py", line 43, in start
> > 	    raise "invalid maildir '%s'" % self.basedir
> > 	invalid maildir '/home/ed/BuildBot/cvsmail': None
> 
> That exception means that the FCMaildirSource wasn't able to read a directory
> named "/home/ed/BuildBot/cvsmail/new". It could be that the directory doesn't
> exist, or that the permissions aren't set up to allow the user running the
> buildmaster read from it.
> 
> If you are running buildbot-0.4.0, you can just manually insert the base
> directory to work with.. something like:
> 
>  source = FCMaildirSource("/home/ed/BuildBot/cvsmail"),
>                           prefix="netcdf-3")
> 
> the os.path.join is just a convenience: it lets you relocate the entire
> buildmaster by rebuilding the .tap file with a different --basedir argument,
> without having to modify the master.cfg file as well.
> 
> Let me know if that helps or if you run into problems..
>  -Brian