[users at bb.net] 0.9.0rc2, multi-master, and anecdotes on reconfig.
ngilmore at grammatech.com
Thu Nov 3 21:27:11 UTC 2016
Good afternoon everyone,
It seems that after a branch change and reconfig, our buildbot system is
misbehaving. Builds are not getting scheduled.
In our system, when we start a new branch in svn, we update a line in
our master.cfg. This results in new builders and schedulers being
created that use the new branch name when we reconfig.
I stuck in a little debugging code to dump out the name and builderNames
for the schedulers we're putting into c['schedulers']. It appears as
though the builders for the new branch are in the proper schedulers. We
have 4 masters, but these schedulers only appear in the 3 non-UI
masters. The UI master keeps all the force schedulers.
But though the builders appear in the UI, they never get scheduled.
Certainly the hourly ones aren't.
So into the manhole for some spelunking. Looking at
master.scheduler_manager.namedServices[<name>], I see only 2 builders.
This is so in the 2 masters I looked at. I'm informed that these 2
builders were moved to this new branch some time ago, possibly before
the multi-master move.
The appearance is that reconfig is not actually reconfiguring correctly.
Either it isn't killing the current scheduler and using a new one
created from master.cfg info, or it's not putting the new master.cfg
info into the existing scheduler.
I have had reports of other vague reconfiguration problems, but nothing
I'd write an anecdote about.
As far as I know, this was not a problem with our old 0.8.x system using
a single master, but that predates me.
I may attempt to kill off that scheduler via the manhole, then reconfig.
If it comes back with all the proper builders, that'll be a clue, won't it?
More information about the users