[users at bb.net] More anecdotes, from 0.9.3 multi-master this time.

[Neil Gilmore]

Hi everyone.

Well, the restart and new database went pretty well. The UI is a lot 
snappier, and I haven't seen any database integrity problems. I will 
have to keep an eye on the crossbar if it ever goes down, because I 
don't quite trust cron to start it correctly. My redirected stdio showed 
some weirdness, but I didn't keep the log.

But there's been something new. We appear to be getting multiples of the 
same build in the queues for some builders. One builder might show 3 
builds in its queue, 2 submitted at the same time, one a few seconds 
later. Otherwise, they're indistinguishable from each other. We weren't 
getting this before. I have build collapsing turned on, and we have a 
modification to the code to not consider revision when collapsing 
(always use the latest code). But if collapsing were actually working, 
I'd expect for those builds to be collapsed (unless... see below).

My immediate thought was that the builds were getting scheduled by all 3 
masters. But that shouldn't be. A while ago, I changed out master.cfg to 
append the name of the master to each scheduler. This meant that we 
wouldn't get any failover if a master failed. But it also meant that 
whichever master started first didn't get all the active schedulers.

One other oddity is that the name of the scheduler in the build UI does 
not match any name in the schedulers list. If I look at the list of 
schedulers, every scheduler (except for the force schedulers) has the 
name of its master appended. If I look at the build's properties, I see 
the name without the master's name included. The only oddity I see is 
that I create the scheduler first, then add the master's name to the 
scheduler's name just before adding it to the schedulers.

I'm not seeing anything I'd take as unusual in the logs, but I'm not 
such a good log whisperer. The only place the schedulers seem to be 
named in the logs are from some debug info I put in to keep track of 
what ended up in which master.

Neil Gilmore
grammatech.com