[Buildbot-devel] reloading config says nothing changed

[Brian Warner]

> If I go to reload my config, after changing my "is file important?" 
> checks for example, the build master log says nothing changed so 
> nothing happens.  What is the master checking against to determine 
> whether or not the config changed?

The short answer is that the Reload Config button (in the debug tool) only
looks at changes to the configuration (the keys and values of the
BuildmasterConfig dictionary), and doesn't know anything about the source
code that implements the objects described therein.

There is another button for reloading source code modules (named Rebuild),
but it doesn't do anything yet. Twisted provides fairly sophisticated support
for reloading a module and then migrating all existing objects to the new
code, but I haven't figured it out enough to use it yet.

The workaround is to stop your buildmaster (SIGINT), then restart it from the
saved .tap file with something like:

 twistd -f buildbot-shutdown.tap

The shutdown will pickle (serialize) the entire application out to the file,
then the reload will pull it all back in (but using the new code). It's more
traumatic than a real rebuild should be (i.e. slave connections are lost and
must be reestablished), which is why I plan to move to twisted.python.rebuild
eventually, but at this point it shouldn't lose any state.


The longer answer is that the BuildFactories are compared for equality in
master.BuildMaster.loadConfig_Builders . In particular, the Builder is
considered equivalent (and therefore not touched) if .slavename, .builddir,
and .buildFactory are all equal. The first two are strings, the last is an
instance, which means the __cmp__ attribute is used to determine equivalence.
The default buildbot.process.base.BuildFactory.__cmp__ method compares
classnames and the contents of the instances' __dict__ (instance variable)
dictionary. This should be enough to capture any differences that get stashed
in an attribute somewhere.

So if there were a flag in your BuildFactory like .ignoreHTMLfiles, and you
set that flag based upon one of the arguments to the factory's constructor,
and you changed your config file to start setting the flag, then a "Reload
Config" *would* notice that the factory was changed, and would then remove
the old Builder and replace it with the new one. However, if you changed the
BuildFactory's (or the Builder's) code to use the new flag, A) the Reload
Config button wouldn't even see the .py file got changed and would keep using
the code that it loaded back at startup time, B) if you stop and restart the
buildmaster then the old pickled instance would be loaded against the new
code, which might cause problems (remember that __init__ is *not* re-run, so
your code may reference an attribute that your instances don't actually
posess, resulting in an exception).

Handling code changes on the fly is tricky. Twisted provides an entire class
to make it easier (twisted.python.styles.Versioned). Basically you need to be
aware every time you add or remove an attribute, and provide for the case
that the attribute might not be present in a saved instance.

I run into cases all the time (well, every few months) with the Twisted
buildbot where I just have to shut down and launch a new copy, losing all of
the event history, because I'm too lazy to do all the necessary compatibility
work.

hope that helps,
 -Brian