[users at bb.net] More odd behavior.anecdotes for multi-master.

[Neil Gilmore]

Hi all,

Work on making schedulers reconfigurable is progressing, though slowly 
as I find myself again having to rectify troubles in our live system.

Today, we have some builders in an odd state (I haven't gone though the 
whole list yet). For one particular builder, looking at the builder page 
shows that the current build is a retry, started 2 days ago, has a 
duration of 5 hours, and has a status of 'building'. Buildrequests are 
queuing up, presumably because of the status. Clicking on that build, 
however, indicates that the build finished 2 days ago. Cancelling the 
queue does not finish the current build. Forcing a build worked, though 
the build failed for unrelated reasons (I think something needed 
cleaning up that doesn't happen when I manually finish some builds).

I had to manually finish 55 builds.

We also have more than one builder that claims to have 2 retrys building 
at the same time, though going through the manhole, 
master.botmaster.namedServices[<name>].building only shows one building.

And that's today's news.

Neil Gilmore
grammatech.com