[Buildbot] #3599: improve multimaster documentation

[Buildbot trac]

#3599: improve multimaster documentation
-------------------+-------------------
Reporter:  tardyp  |      Owner:
    Type:  task    |     Status:  new
Priority:  major   |  Milestone:  0.9.1
 Version:  master  |   Keywords:
-------------------+-------------------
 basically need to rephrase this and put it in doc:

 In nine, there is the new concept of clustered service, which are service
 that runs on only one master, masters are competing to run those service,
 and the first master which will claim this service will run it. schedulers
 and changesource are all clustered services.
 The database will act as an arbitrator (hence multimaster cannot work with
 sqlite)
 https://github.com/buildbot/buildbot/blob/master/master/buildbot/db/model.py#L254


 What is not implemented is load balancing between master. Basically if you
 run a symetric multimaster configuration(as per concepts.rst), the first
 master that will start will take all the schedulers and change sources.

 If that first master stops normally, then the second master will take
 over.
 So basically high availability is present (with the slight delay of second
 master reacting to first master shutdown event), but not load balancing.

 While writting this email, I am already finding some issues. If the first
 master is crashing, and not sending a ('master', masterid, 'stopped')
 event, then there is a mecanism of keep-alive timeout, and second master
 will notice the first master is dead only after this timeout.
 Means that during that time the schedulers and change source will not
 update anything.

 The forcescheduler though shall not be impacted by this problem.
 Forceschedulers have to be configured on the same masters as the web
 servers, and the webserver will use the local forcescheduler to trig their
 builds, even if it is not technically started as a clustered service.
 Again, that is in theory, and this has not really been tested.


 also need to fix this as it is based on eight design:
 http://docs.buildbot.net/latest/manual/cfg-global.html#multi-master-mode

--
Ticket URL: <http://trac.buildbot.net/ticket/3599>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation