[users at bb.net] 0.9.0rc2, multi-master, and anecdotes on reconfig.

[Neil Gilmore]

Good afternoon everyone,

Another anecdote.

It seems that after a branch change and reconfig, our buildbot system is 
misbehaving. Builds are not getting scheduled.

In our system, when we start a new branch in svn, we update a line in 
our master.cfg. This results in new builders and schedulers being 
created that use the new branch name when we reconfig.

I stuck in a little debugging code to dump out the name and builderNames 
for the schedulers we're putting into c['schedulers']. It appears as 
though the builders for the new branch are in the proper schedulers. We 
have 4 masters, but these schedulers only appear in the 3 non-UI 
masters. The UI master keeps all the force schedulers.

But though the builders appear in the UI, they never get scheduled. 
Certainly the hourly ones aren't.

So into the manhole for some spelunking. Looking at 
master.scheduler_manager.namedServices[<name>], I see only 2 builders. 
This is so in the 2 masters I looked at. I'm informed that these 2 
builders were moved to this new branch some time ago, possibly before 
the multi-master move.

The appearance is that reconfig is not actually reconfiguring correctly. 
Either it isn't killing the current scheduler and using a new one 
created from master.cfg info, or it's not putting the new master.cfg 
info into the existing scheduler.

I have had reports of other vague reconfiguration problems, but nothing 
I'd write an anecdote about.

As far as I know, this was not a problem with our old 0.8.x system using 
a single master, but that predates me.

I may attempt to kill off that scheduler via the manhole, then reconfig. 
If it comes back with all the proper builders, that'll be a clue, won't it?

Neil Gilmore
grammatech.com