[Buildbot-devel] trying to use buildbot for freeware scientific data package...

[Brian Warner]

> Now I am going through the chore of installing everything on our many
> different types of machines. Mostly I have to upgrade python first, as
> many systems here don't have a very recent version.

FYI both Twisted and Buildbot should work fine with python-2.2, so it doesn't
have to be the very very latest version. I generally test again python-2.3,
but it ought to work with 2.2 as well. I know for a fact that Twisted works
ok with python-2.2.0, because we've got a buildbot slave running that test
suite.

> This is difficult in some cases. For example, on one system, Twisted
> seemed to install just fine, but when I try and do:
> 
> % twistd -d buildbot.tap 
> 
> I get an error:
> 
> Failed to load application: [Errno 2] No such file or directory:
> 'twistd.tap'

Hm, that's not the right argument. Read on..

> Where does twistd.tap really live? I don't know yet...

The .tap file is created when you run 'mktap', in whatever directory you were
sitting in at the time. 'mktap' picks a name for the .tap file based upon the
kind of .tap you are creating: when doing 'mktap buildbot master --basedir
foo', it creates 'buildbot.tap'.

I usually move that buildbot.tap file to the base directory itself, and then
launch it with a short Makefile that looks like this:

---------- START ----------
BBBASE = ~/stuff/Projects/BuildBot/sourceforge
PP = PYTHONPATH=$(BBBASE):~/stuff/python/twisted/Twisted-CVS-RW:.
#PP = PYTHONPATH=$(BBBASE)

start:
	$(PP) twistd -f buildbot.tap
stop:
	kill `cat twistd.pid`
restart:
	$(PP) twistd -f buildbot-shutdown.tap
log:
	tail -f twistd.log
---------- END ----------

You want the -f option to specify which .tap file to run, not the -d option
(which specifies a directory to switch to: unnecessary because the buildbot
does that itself based upon the recorded --basedir argument).

The $(PP) option is a simple hack to let me put the development version of
buildbot (or of Twisted) on my PYTHONPATH. You should take this out, or you
could use it to run against a copy of buildbot or twisted that isn't
installed in /usr/lib/ .

With this Makefile, I just do:

 make start

to fire off the daemon. Then I do 'make log' to watch it's output. Sometimes
I use a target like this:

run:
	$(PP) twistd -l - -n -f buildbot.tap

to make twistd *not* fork into the background. This is handy when my
development code is crashing at startup and I want to see everything that
happened (instead of starting it, watching it fork into the background,
knowing it will die in a moment, waiting for that, then looking at twistd.log
to see what went wrong). The "-l -" means "log to stdout instead of
twistd.log", and the "-n" means "don't fork into the background".

I don't use that 'make run' target for production systems because when you
ctrl-c out of it, the buildmaster shuts down (because of the -n). I only use
it for quick tests.

This makefile makes it easier to set up a @reboot cron job to start the
daemon automatically at boot time.

 @reboot make -C ~/path/to/basedir start

This is how the Twisted buildmaster and most of the slaves are started each
time their host reboots. The only thing to keep in mind is that the
environment provided to cron jobs may be different than the one in your
normal shell (typically a different set of .bashrc/.bash_profile scripts are
run). You may need to do some experimentation.

cheers,
 -Brian