[Buildbot-devel] trying to use buildbot for freeware scientific data package...

[Brian Warner]

> From: Ed Hartnett <ed at unidata.ucar.edu>
> 
> Thanks for your reply yesterday! It was very helpful, and now I
> believe (from the log files) that I have BuildBot up and running as a
> master and as a slave, and that they are communicating. For example,
> from my master log:
> 
> 2004/01/08 11:56 MST [-] Log opened.
> 2004/01/08 11:56 MST [-] twistd 1.1.1 (/usr/bin/python 2.2.3) starting up
> 2004/01/08 11:56 MST [-] reactor class: twisted.internet.default.SelectReactor
> 2004/01/08 11:56 MST [-] Loading buildbot.tap...
> 2004/01/08 11:56 MST [-] Loaded.
> 2004/01/08 11:56 MST [-] set uid/gid 4178/2000
> 2004/01/08 11:56 MST [-] loading configuration from /home/ed/BuildBot/master.cfg
> 2004/01/08 11:56 MST [-] opening /home/ed/BuildBot/cvsmail/new
> 2004/01/08 11:56 MST [-] adding new builder netcdf-3-c-only
> 2004/01/08 11:56 MST [-] twisted.spread.pb.PBServerFactory starting on 8007
> 2004/01/08 11:56 MST [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x853e884>
> 2004/01/08 11:56 MST [-] BuildMaster listening on port 8007
> 2004/01/08 11:56 MST [-] checking web port 8080
> 2004/01/08 11:56 MST [-] adding web port
> 2004/01/08 11:56 MST [-] twisted.web.server.Site starting on 8080
> 2004/01/08 11:56 MST [-] Starting factory <twisted.web.server.Site instance at 0x8513684>
> 2004/01/08 11:56 MST [-] configuration updated
> 2004/01/08 12:10 MST [Broker,0,127.0.0.1] Peer will receive following PB traceback:
> 2004/01/08 12:10 MST [Broker,0,127.0.0.1] Failure: twisted.cred.error.UnauthorizedLogin: 
> 	
> 2004/01/08 12:57 MST [Broker,1,127.0.0.1] returning Avatar(EdsLinuxBox1): <BotPerspective 'EdsLinuxBox1', builders: netcdf-3-c-only>
> 2004/01/08 12:57 MST [Broker,1,127.0.0.1] bot attached
> 2004/01/08 12:57 MST [Broker,1,127.0.0.1] Builder netcdf-3-c-only attached
> 2004/01/08 12:57 MST [Broker,1,127.0.0.1] Got slaveinfo from 'EdsLinuxBox1'
> 
> Which seems to indicate that the slave EdsLinuxBox1 is up and running,
> and connecting with the master.

Yes, everything looks good. It appears that your buildslave used the wrong
password once (that "UnauthorizedLogin" line) but that then you fixed it and
it was able to connect successfully.

> Next problem is setting up the CVS notification. I am wondering how to
> tell CVS to notify me by email. At first I thought it would be the
> watch command, but that seems to turn on the read-only flag for all
> watched files, and that's not how we work here.
> 
> In any case, I still can't get CVS to send me (or anyone else) email
> when the repository changes. (Yes, I am using two users, for CVS won't
> notify one if his own changes, only someone else's).

You'll want to modify the "loginfo" file in the CVSROOT directory. That file
lets you run a program every time someone (anyone) checks in a file. Take a
look at the (loginfo) node of the CVS Info documentation (Administrative
Files / commit support files / loginfo).

Look for a package called 'syncmail', which (like FreshCVS) installs itself
into your repository's CVSROOT/loginfo, and uses that hook to send out mail
when a file has been checked in. FreshCVS gives you a quicker response (in
fact the notification is delivered to the buildmaster even before the
committing user sees it), but something which only sends out mail may be
easier to get set up.

The "watch" command is just for individual files, watched by individual users
(to sort of claim ownership over the file), and requires that you work in the
reserve/edit/commit mode that is not the usual CVS style. loginfo is the way
to go.

> Right now I am pouring through the CVS documentation to get some
> notification email. Meanwhile, is there any way to trigger the slave
> to build the package and test it? For example, is there some command I
> can send it which will cause that to happen?
> 
> Right now a manual trigger would at least give me some action, while I
> try and figure out CVS.

If you load up the buildmaster's status web page, you'll see some links at
the top with the name of each Builder. Follow that link to get to the Builder
status page, and you'll find a pair of text fields and a button labeled
"Force Build". Fill in the text fields with your name and a reason for the
build (these are put into the status output to replace the usual list of
Changes that triggered the build) and push the button. That will kick off a
build immediately.

There is also a flag you can set from the config file which will cause a
build to be scheduled every N seconds, which might be useful (if somewhat
wasteful of CPU time and CVS bandwidth) until you get your CVS notification
up and running.

Also, if you run the debug tool (contrib/debugclient.py), there are some
buttons there to force builds by name.

> And if you have any knowledge about setting up CVS to send
> notifications, that would be helpful.
> 
> Thanks for all the help! I feel like this is close to working for me
> and I'm very excited, as it will save me weeks and weeks of work over
> the next year.
> 
> Ed Hartnett
> Unidata

Glad to hear it!

> PS - I also tried to install CVSToys, but after modifying my CVS admin
> files is crapped out and never worked. So I had to go back and
> manually restore the CVS admin files to stop annoying error messages
> which pop up for every CVS action. Fortunately all this took place at
> 5 am, so I had plenty of time to fix it before the other developers
> came in and worried about be messing up the CVS repository. 
> 
> CVSToys should come with a big warning attached!

Well, usually it works, and the admin files it modifies are the same ones
you'll be modifying to get mail notification happening. The messages *are*
annoying, but they're benign (CVS is smart enough to survive errors in the
loginfo scripts).

What I did was to create a new dummy CVS repository and use it to test out
FreshCVS, so I could figure out how it worked without affecting up my real
Repository until I was ready. That let me track down a few PYTHONPATH issues
(FreshCVS needs Twisted too) and gave me a chance to figure out the freshCfg
config file syntax.

That said, buildbot+freshcvs(pb) are in the process of making transitions
from Twisted's old authentication scheme to the new one, so there are cases
where things might break. The Twisted project's CVS repository's CVSToys
installation currently requires manual intervention (using a debugging tool
called "manhole") to get the buildbot's notification port opened up, so every
time that machine is rebooted the buildbot loses its connection. That is why
we fell back to using mail-based notification (with noticibly higher latency,
sometimes 10-15 minutes).

Once you get it set up, CVSToys is really nice, and works exceptionally well
with the Buildbot (where the immediate feedback is key to supporting the
buildbot's goals: help developers fix their problems before they impact
anyone else).


cheers,
 -Brian