[users at bb.net] 0.9.0rc2 and multi-master

[Neil Gilmore]

Hi Pierre,

As always, thanks for steering me in the right direction.

First off before we get to the interesting stuff, a couple nits in the 
documentation...

Here 
(http://docs.buildbot.net/latest/manual/cfg-global.html#mq-specification) 
I see:

c['mq']  =  {
     'type'  :  'wamp',
     'router_url':  'ws://url/to/crossbar'
     'realm':  'buildbot'
     'debug'  :  False,
     'debug_websockets'  :  False,
     'debug_lowlevel'  :  False,
} It would be very, very helpful to those doing a copy/paste on that to 
include commas after the crossbar URL and the realm. It would also be 
useful to include that unicode indicator before 'buildbot'. It might 
also be helpful to add host:port to the url, just so the user doesn't 
forget to put it in there.

Now on to the good stuff.

On 9/6/2016 9:18 AM, Pierre Tardy wrote:
> You need to configure all the force-schedulers on your UI masters.
> forceschedulers defined in the non-UI masters will not be used, as 
> they have to be in the UI.
> This is tracked on http://trac.buildbot.net/ticket/2673

This is good to know, and not hard to do. Most of my users want to be 
able to force builds.

> the masters are talking to each other via crossbar. an easy way to 
> make sure this work is to start a build on a non-ui master, and see it 
> auto-update in the ui-master.

While I was not using symmetric masters, I'm certain I got this working, 
even though I was doing the force in the UI master.

> If you get autoupdate then the mq is correctly working. If you have to 
> reload the UI (F5) to see build progress, then it means the ui master 
> can only get updated data via DB, and not via message queue.
> You might also have a look at tcpdump on the crossbar port to see the 
> activity to crossbar.

I think it's sufficient to run crossbar in the foreground using a 
loglevel of debug. Since each master has a different signature, you can 
see them talk. Probably a bit easier (at least for experimentation) than 
using tcpdump.

> I would recommend to first start with symetric configuration where all 
> the masters have UI, and all masters have all builders/schedulers

I can see where it might be easier to test things this way, but the 
whole point for us to use multi-master is to NOT have all 
builders/schedulers active on the same masters. For our purposes, it'll 
be bad enough that the database and crossbar are single points of failure.

I still see what I interpret as 'lost deferred' problems often, and I 
think we're leery of not being able tor recover from them easily in a 
way that doesn't inconvenience people by having to take down a master 
controlling a build that they're waiting for. Every time a deferred is 
lost and the master effectively stops talking to the worker, it's 
unpleasant. And it happens pretty often for us.

Anyway, I have asymmetric multi-master working in a small test setup 
just using mostly the defaults and a postgres database (as I'm not 
familiar with MySQL). I may even be able to create that tutorial you'd 
like (but no promises).

Now I just have to get it working with our master.cfg. I think the plan 
is to use the same master.cfg for every master, but have it process 
differently for each master. Part of the idea I have is that if someone 
needed to, they could run their own master fairly easily, tied in to the 
overall system. We definitely plan to have the UI as separate as 
possible, as well as a couple other things that people tend to complain 
about when they aren't running.

Neil Gilmore
grammatech.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20160907/1eb53e32/attachment.html>