<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Pierre,<br>
<br>
As always, thanks for the reply.<br>
<br>
Just to be perfectly clear, most of the recent troubles weren't with
buildbot itself. It's more my unfamiliarity with the Python
ecosystem. We are , for various reasons, building everything from
source distributions. so docker or ansible isn't really what we're
doing. What I have is a straight-up bash script. Not a great one,
though.<br>
<br>
Thanks for the ideas. Changing the name might work in the short run.
Not a good long-term solution for us, as we have a specific naming
convention to keep people from going crazy.<br>
<br>
The problem in this case isn't that the worker won't start or
complete builds, which isn't quite the usual problem. It's that we
have a builder that claims 2 builds are in progress. The most recent
is acquiring locks because the other builder on that worker is
building. It's the one behind that that claims to still be building,
even though it's obviously stalled.<br>
<br>
The same builder picked up the lock 2 builds in a row. I seem to
recall you saying that's possible. In this case, we do need the
output of the screwed-up builder, so I really can't wait for another
3-day build. Looks like it'll get moved to the alternate for now.<br>
<br>
It's good to know that the problem is probably in the python rather
than in the database.<br>
<br>
Some figures from the impending multi-master move:<br>
master 1:<br>
workers: 164<br>
builders: 355<br>
schedulers: 80<br>
master 2:<br>
workers: 27<br>
builders: 87<br>
schedulers: 80<br>
master 3:<br>
workers: 2<br>
builders: 6<br>
schedulers: 80<br>
master 4:<br>
workers: 0<br>
builders: 0<br>
schedulers: 450<br>
<br>
Masters 1 and 2 represent the two major divisions of labor that we
have. Master 3 contains those builders that people notice and
complain about most when they fail or are not present, even more
than the ui. Master 4 contains only the ui and force schedulers.
Since you said that the schedulers behave regardless of how many
masters they are on, I left them on all the masters. At some point,
I expect to split master 1 up some. And I'll be providing guidance
to those here who might want to set up a semi-private master for a
specific project.<br>
<br>
People here expect to see the UI on a specific URL, so I'll end up
bringing up multi-master on its own, the reconfiguring to point the
UI master to where it ought to be.<br>
<br>
Neil<br>
<br>
On 9/29/2016 3:54 AM, Pierre Tardy wrote:<br>
<blockquote
cite="mid:CAJ+soVdvLt9R4OFj0DoLwa_qQ+=ARTt+Vyq9j76avpBuD9hnqQ@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Neil,
<div><br>
</div>
<div>Good to know your multimaster setup is nearly done!</div>
<div>Indeed scripted multimaster would be great!</div>
<div>docker-compose setup or ansible playbook would be perfect I
think</div>
<div><br>
</div>
<div><br>
</div>
<div>For the other part of the annecdote I think we are still
chasing the same bug right.</div>
<div>At some point somehow, you got a worker that is in a bad
state and won't accept anymore build.</div>
<div><br>
</div>
<div>The idea to move to master is a workaround indeed. Another
one is to change the name of the worker.</div>
<div><br>
</div>
<div>Obviously the best long term option is to debug the
problem, but it does n't look like easy to reproduce nor to
debug :-(</div>
<div>hacking the database? naah.. dont. really.</div>
<div>It wont even help as most likely the corrupted state is in
the python objects</div>
<div><br>
</div>
<div>Pierre</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Le mer. 28 sept. 2016 à 22:33, Neil Gilmore <<a
moz-do-not-send="true" href="mailto:ngilmore@grammatech.com">ngilmore@grammatech.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi everyone,<br>
<br>
Congrats on rc4.<br>
<br>
More anecdotes from rc1. I got tangled up a bit trying to get<br>
multi-master working. I'm still not sure why all the parts
would build<br>
one day, then not the next (in this case, it was setuptools).
Nor why<br>
crossbar requires libffi to be installed on one machine but
not the<br>
other. Nor why SQLAlchemy will be downloaded and installed
automatically<br>
but not psycopg2. These troubles seem to have straightened
themselves<br>
out, and I have multi-master buildbots in sandboxes on 2
different<br>
machines. There's light at the end of the tunnel, I hope.<br>
<br>
As side note, Pierre, I ended up scripting the whole
install/build/run<br>
thing. That may have to do for a tutorial.<br>
<br>
I got asked for help with a builder. Seems it was taking
inordinately<br>
long to do a build, and the user tried cancelling, forcing,
etc. There's<br>
3 builders for this worker. 1 doesn't use locks, but the other
2 do.<br>
It's pretty common for our workers to have a builder that
doesn't lock,<br>
and the rest do.<br>
<br>
The current situation is that the build of the builder in
question shows<br>
not 1, but 2 builds building. Sort of, the current build is
shown as<br>
acquiring locks. The older building build is clearly stalled.<br>
<br>
The other builder for the worker is proceeding well (but its
builds take<br>
about 3 days). Obviously, it was able to get the lock. But it
has<br>
started another build after finishing the first one. So it
appears that<br>
it got the lock again before the original builder (unless
there's<br>
something else going on).<br>
<br>
I also had a different worker's build stall, so I moved that
worker to<br>
our alternate master. Unfortunately, it's a trick that only
works once.<br>
If I move it back, it'll still be stalled. Is there any way to
remove a<br>
no longer active worker from the database? I tried once, but I
messed it<br>
up and had to start with an empty database. I didn't try
again.<br>
<br>
Neil Gilmore<br>
<a moz-do-not-send="true" href="http://grammatech.com"
rel="noreferrer" target="_blank">grammatech.com</a><br>
_______________________________________________<br>
users mailing list<br>
<a moz-do-not-send="true" href="mailto:users@buildbot.net"
target="_blank">users@buildbot.net</a><br>
<a moz-do-not-send="true"
href="https://lists.buildbot.net/mailman/listinfo/users"
rel="noreferrer" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><br>
</blockquote>
</div>
</blockquote>
<br>
</body>
</html>