[users at bb.net] More from the land of multi-master.

[Neil Gilmore]

Pierre,

As always, thanks for the reply and advice.

Note that I've clipped items that were addressed and that I have no more 
comments on.

On 10/5/2017 3:45 AM, Pierre Tardy wrote:
>
>
> On Wed, Oct 4, 2017 at 5:12 PM Neil Gilmore <ngilmore at grammatech.com 
> <mailto:ngilmore at grammatech.com>> wrote:
> We have also been getting a lot of errors apparently tied to build
>
>     collapsing, which we have turned on globally. If you've been following
>     along with the anecdotes, you'll know that we've also slightly
>     modified
>     the circumstances under which a build will be collapsed to ignore
>     revision (in our case, we always want to use the latest -- we
>     don't care
>     about building anything 'intermediate'). We'd been getting a lot of
>     'tried to complete N buildequests, but only completed M' warnings.
>
> We have seen also people seeing those issues. I have made a fix in 
> 0.9.10, but it looks like there are still people complaining about it, 
> but without much clue of what is wrong beyond what was fixed.
> The known problem was that the N buildrequests were actually not 
> uniques buildrequests, the list contained duplicated.
> So those warnings should be pretty harmless beyond the noise.

Even though the transaction involving those buildrequests cancels the 
transaction, so that the original work of marking requests doesn't 
happen? Or would that just mean the requests don't get skipped?

It does seem like the incidence of this warning has been going down, 
though we haven't done anything to fix it.

>     And I
>     left some builders' pages up in my browser long enough to see that
>     every
>     build (except forced builds) was getting marked as SKIPPED eventually.
>     Forced builds were never getting claimed. Nor were the skipped builds
>     marked as claimed, which is odd, because the collapsing code claims
>     builds before marking them skipped. And the comments indicate that a
>     prime suspect in getting that warning is builds that were already
>     claimed.
>
> Normally the buildrequest collapser is not supposed to mark *builds* 
> skipped. It marks buildrequests as skipped.
> So could that be another thing in your steps?

My mistake in using the wrong term here. The code appears to claim the 
request then mark it as skipped. But in the UI, I never see a skipped 
request marked as claimed.

>
>     The result of this is that our master is failing in its prime mission,
>     which is to run builds. I've been occasionally able to get a build to
>     happen by stopping the worker. When our process starts the worker back
>     up, and it connects, the master will look for a pending build and
>     start
>     it. But any subsequent builds will not start. And if there aren't any
>     queued builds, a build that gets queued while the worker is running is
>     not started. And the builder we use to start workers, which is
>     scheduled
>     every half hour, didn't run for 18 hours (though it seems to have just
>     started a build).
>
> Not sure exactly how to answer to that. This is not normal, but there 
> are many reason which could be leading to that situation.
> in my experience, very often it is related to some customization code 
> that is failing.
> Is the first build correctly finished?, is there a nextWorker that is 
> not behaving correctly, do you have custom workers?
> I've seen people having good results by using Manhole to debug those 
> freezes.

The only actual custom code we have is a pair of custom build steps that 
produce logs useful to us, and the modification to collapsing to ignore 
revision.

The builder we have to start workers does not use the custom steps, 
though we have collapsing turned on globally. I have not seen that 
builder having any skipped requests. It appears to be running normally 
since yesterday.

For the builder that only wants to run once, the first build finishes 
correctly.

We do not have custom workers.

> https://docs.buildbot.net/current/manual/cfg-global.html#manhole
> That could help you pinging into the workers and workerforbuilders 
> objects looking for their states
>
>

I've used the manhole before, but not for this. I've had to use it in 
the past to manually finish stuck builds, and to manually release locks 
when necessary (though I haven't had to do that in a long time).

But we don't leave the manhole open, which means that I reconfig when 
I'm going to use it (and since we use the same master.cfg for all the 
masters, the manhole would try, and probably fail, to open for all of 
them). Lately, that hasn't been a good option, because when we were 
having the CPU spikes, the reconfig would never finish (it might run for 
24 hours or more until we were going to restart the master anyway). It 
might work now, though, since we seem to have solved the CPU problem.

Neil Gilmore
grammatech.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20171005/69a40cc6/attachment.html>