[Buildbot-devel] Re: problems with latest version of buildbot

[Brian Warner]

> I just installed the latest version of buildbot, but am having some
> problems...
> 
> First, a few background questions:
> 

> 1 - Do I have to rerun the mktap program to get new buildbot.tap files
> on the slaves when I install the new buildbot?

> 3 - Do I have to rerun mktap on the master?

The answer is "maybe".

In Twisted, .tap files contain both the serialized current state of the
program and a representation of its mktap-time configuration. (using them to
hold current state is vaguely discouraged, but it's so darned convenient that
I haven't yet fixed the buildmaster to do something better).

In an ideal world, the Twisted load-time upgrade mechanism (styles.Versioned)
means that you only ever create the .tap file once, when you originally
install the first version of the package, with a process like this:

 mktap buildbot OPTIONS   # creates buildbot.tap
 twistd -f buildbot.tap   # launch daemon, taking initial state from the .tap

Later, when a new version is available, the upgrade process is:

 shutdown the application (send it SIGINT)

 wait for it to write out the buildbot-shutdown.tap file. This is a
 serialized form of the daemon's state when it received the SIGINT.

 install the new code (setup.py install)

 use 'twistd -f buildbot-shutdown.tap' to re-launch the daemon. This
 rejuvenates the old daemon, performs any object upgrades necessary, and
 restarts all services.

If everything works well, this upgrade should occur with no loss of
application state. Note that the exactly same process (without the 'setup.py
install') can be used to, say, shut down the daemon for a system reboot, then
restore it without data loss after the OS is restarted.


Now, mind you, that's the theory. In practice, it's quite tricky to get the
upgrade path right. I've made some attempts to get one-version-to-the-next
upgrades to work correctly, but I've found it very very difficult to test. So
my general approach is:

 shutdown
 upgrade code
 launch the -shutdown.tap file and see what happens
  if that fails, mktap a new .tap file and launch it instead

Worst case, you lose the historical data (the waterfall display). I've got a
TODO item to move this data out of the .tap file so it won't be lost when you
re-mktap.

I haven't seen any problems with the non-upgrade shutdown/restart sequence.
It's just the shutdown/upgrade/restart that is hard. But give it a try, I
think I have the right code in place to make it work. If you see an exception
in the logs, let me know so I can try to fix it.

That said, there is very little configuration data in the buildmaster (just
the location of the master.cfg file), and the buildslave has almost no
runtime state to speak of. On the slaves, you probably don't have to re-mktap
ever. On the master, you only need to do it if the automatic-upgrade fails.

Also note that the slaves and the master don't necessarily have to be running
the same version. I try to avoid requiring strict synchronization between
them (usually it's only new features like slave-side SVN support that would
require a slave upgrade).

> 2 - What version of twisted is required?

The README says Twisted-1.1.0, and we're running 1.1.1 on the Twisted
buildbot. I think it ought to work against the current release (1.3.0) but I
haven't had a chance to test it myself yet.


hope that helps,
 -Brian