[users at bb.net] rc2 We have acheived multi-master. Mostly.

[Neil Gilmore]

Good morning everyone,

This week I moved over to multi-master. It's kind of working.

We have 4 masters. One just has the UI and force schedulers (and the 
other schedulers, too). One has a particular category of workers. One 
has the other category of workers. And the last is supposed to have our 
process for monitoring results.

It mostly works. Much of what doesn't is most likely my fault. For 
example, it turned out to be very much easier when porting our 
master.cfg to multi-master to decide whether to add a builder after 
creating it rather than before. Unfortunately, I seem to be adding 
MailNotifiers whether or not the builder is added. The result is that 
every failed build results in 3 emails.

That would have been merely comical, except that a process that nearly 
every builder uses (reporting stuff) crashed sometime this morning, 
which resulted in nearly every build failing (couldn't report). And 
other glitches in the conversion of master.cfg prevented the crashed 
process from getting restarted.

At least I can fix that easily enough.

We do get an awful lot of 
buildbot.db.schedulers.SchedulerAlreadyClaimedError. I'd left all the 
schedulers on all the masters as Pierre indicated to me that if one 
master fails, another will pick up the scheduler. But it seems as though 
this ought to be one of the exceptions suppressed by c['multiMaster'] = 
True.

And congratulations on getting 0.9.0 released!

Neil Gilmore
grammatech.com