[devel at bb.net] Multi-master with 3 masters

[Neil Gilmore]

(didn't see the original message, sorry about that)

Aakash,

We use a very similar setup and for similar reasons. Our single-master 
setup was bogging down completely. However, we elected to not put any 
schedulers except our force schedulers into our ui master. In our setup, 
every builder is tied to a particular master, so the separation came 
naturally. We found it better for us to duplicate essentially identical 
schedulers under different names (each scheduler having the master that 
owns it as part of the name).

We also use a single master.cfg to configure each master. Our master.cfg 
uses basedir to determine which master is currently using it (and uses 
'.' if none is, as when doing buildbot checkconfig . for example). But 
ours is pretty complicated.

We also have a script which will do a reconfig of all our masters at the 
same time, and let us know if the reconfig was successful. That way we 
don't have to worry about missing a master when we reconfigure.

One thing to keep tucked into your head is that if a master loses its 
connection to the wamp router, you start getting some odd things 
happening. But it happens so seldomly that it's hard to remember that it 
might be a problem.

Neil Gilmore
ngilmore at grammatech.com

On 9/13/2018 3:15 PM, Pierre Tardy wrote:
> Hi Aakash,
>
> This sounds like a good plan.
>
> What you can have is a haproxy distributing the workerload over your 
> two masters, this helps avoid to specifically configure workers.
> haproxy can work with roundrobin of tcp sockets that what you should 
> configure for your worker master.
>
> Also, you should put your schedulers and changesource in your UI master.
> There are some bugs in the way buildbot do master election, and you 
> shouldn't count on it unless prepared to fix some bugs
>
> You can restart the masters independently without issues afaik.
>
> Regards
>
> Pierre
>
> On Wed, Sep 12, 2018 at 7:43 AM Aakash Jain <aj355 at cornell.edu 
> <mailto:aj355 at cornell.edu>> wrote:
>
>     Hello,
>
>     I have a multi-master Buildbot configuration with following
>     configuration:
>     master1: handling web-server,
>     master2: handling everything else (including workers
>     communication, build-request-distributor etc.)
>
>     Buildbot version 1.1.1
>     Twisted: 17.5.0
>     16-core cpu
>
>
>     However, master2 is probably heavily loaded and I observe large
>     delay in scheduling builds (even when workers are idle)
>     https://github.com/buildbot/buildbot/issues/3661
>
>     I am planning to split master2 into two masters (master2 and
>     master3), one running on pb port 9989 and another one on pb port
>     9990. Some of the workers would be configured to connect to
>     master2 and rest to master3.
>
>     All 3 masters would load same configuration. There wouldn't be any
>     load balancer, since all masters are running on different ports
>     (workers decide which master to connect to based on how they are
>     configured). I am hoping that this would help with the delays,
>     since both master2 and master3 would be running the
>     build-request-distributor (activity_loop).
>
>     - Does this 3-master configuration looks correct?
>     - Is this a supported configuration?
>     - Can I restart one master independent of other?
>     - Any feedback/comment/consideration?
>
>
>     Thanks
>     Aakash
>
> -- 
>
>
>
>
> _______________________________________________
> devel mailing list
> devel at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/devel/attachments/20180913/ac08c3c8/attachment.html>