<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Pierre,<br>
<br>
As always, thanks for steering me in the right direction.<br>
<br>
First off before we get to the interesting stuff, a couple nits in
the documentation...<br>
<br>
Here
(<a class="moz-txt-link-freetext" href="http://docs.buildbot.net/latest/manual/cfg-global.html#mq-specification">http://docs.buildbot.net/latest/manual/cfg-global.html#mq-specification</a>)
I see:<br>
<pre><span class="n">c</span><span class="p">[</span><span class="s1">'mq'</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'type'</span> <span class="p">:</span> <span class="s1">'wamp'</span><span class="p">,</span>
<span class="s1">'router_url'</span><span class="p">:</span> <span class="s1">'<a class="moz-txt-link-freetext" href="ws://url/to/crossbar">ws://url/to/crossbar</a>'</span>
<span class="s1">'realm'</span><span class="p">:</span> <span class="s1">'buildbot'</span>
<span class="s1">'debug'</span> <span class="p">:</span> <span class="bp">False</span><span class="p">,</span>
<span class="s1">'debug_websockets'</span> <span class="p">:</span> <span class="bp">False</span><span class="p">,</span>
<span class="s1">'debug_lowlevel'</span> <span class="p">:</span> <span class="bp">False</span><span class="p">,</span>
<span class="p">}
It would be very, very helpful to those doing a copy/paste on that to include commas after
the crossbar URL and the realm. It would also be useful to include that unicode indicator before
'buildbot'. It might also be helpful to add host:port to the url, just so the user doesn't forget to
put it in there.
</span></pre>
Now on to the good stuff.<br>
<br>
<div class="moz-cite-prefix">On 9/6/2016 9:18 AM, Pierre Tardy
wrote:<br>
</div>
<blockquote
cite="mid:CAJ+soVc9t09hDOxt_JcXrNS3yiHSunxNpRULwUf2g+t7Hfnhdw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_quote">You need to configure all the
force-schedulers on your UI masters.
<div>forceschedulers defined in the non-UI masters will not be
used, as they have to be in the UI.</div>
<div>This is tracked on <a moz-do-not-send="true"
href="http://trac.buildbot.net/ticket/2673">http://trac.buildbot.net/ticket/2673</a></div>
</div>
</div>
</blockquote>
<br>
This is good to know, and not hard to do. Most of my users want to
be able to force builds.<br>
<br>
<blockquote
cite="mid:CAJ+soVc9t09hDOxt_JcXrNS3yiHSunxNpRULwUf2g+t7Hfnhdw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_quote">the masters are talking to each other
via crossbar. an easy way to make sure this work is to start a
build on a non-ui master, and see it auto-update in the
ui-master.</div>
</div>
</blockquote>
<br>
While I was not using symmetric masters, I'm certain I got this
working, even though I was doing the force in the UI master.<br>
<br>
<blockquote
cite="mid:CAJ+soVc9t09hDOxt_JcXrNS3yiHSunxNpRULwUf2g+t7Hfnhdw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>If you get autoupdate then the mq is correctly working.
If you have to reload the UI (F5) to see build progress,
then it means the ui master can only get updated data via
DB, and not via message queue.</div>
<div>You might also have a look at tcpdump on the crossbar
port to see the activity to crossbar.</div>
</div>
</div>
</blockquote>
<br>
I think it's sufficient to run crossbar in the foreground using a
loglevel of debug. Since each master has a different signature, you
can see them talk. Probably a bit easier (at least for
experimentation) than using tcpdump.<br>
<br>
<blockquote
cite="mid:CAJ+soVc9t09hDOxt_JcXrNS3yiHSunxNpRULwUf2g+t7Hfnhdw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_quote">I would recommend to first start with
symetric configuration where all the masters have UI, and all
masters have all builders/schedulers<br>
</div>
</div>
</blockquote>
<br>
I can see where it might be easier to test things this way, but the
whole point for us to use multi-master is to NOT have all
builders/schedulers active on the same masters. For our purposes,
it'll be bad enough that the database and crossbar are single points
of failure.<br>
<br>
I still see what I interpret as 'lost deferred' problems often, and
I think we're leery of not being able tor recover from them easily
in a way that doesn't inconvenience people by having to take down a
master controlling a build that they're waiting for. Every time a
deferred is lost and the master effectively stops talking to the
worker, it's unpleasant. And it happens pretty often for us.<br>
<br>
Anyway, I have asymmetric multi-master working in a small test setup
just using mostly the defaults and a postgres database (as I'm not
familiar with MySQL). I may even be able to create that tutorial
you'd like (but no promises).<br>
<br>
Now I just have to get it working with our master.cfg. I think the
plan is to use the same master.cfg for every master, but have it
process differently for each master. Part of the idea I have is that
if someone needed to, they could run their own master fairly easily,
tied in to the overall system. We definitely plan to have the UI as
separate as possible, as well as a couple other things that people
tend to complain about when they aren't running.<br>
<br>
Neil Gilmore<br>
grammatech.com<br>
</body>
</html>