[users at bb.net] Experiences with two Master Setup 1st web 2nd builds?

[Jan Marker]

Hi,

Posting a follow up: We've now established a multi master setup, with two masters:
1st handles the web interface and force schedulers
2nd handles everything else

It has fixed nearly all our performance issues in the web interface, and after a week we are not experiencing issues.
The Changes API is still the slowest and can occasionally take 10-20 second to respond, but most other APIs have gone from multiple seconds to a few milliseconds.

Thanks a lot for your help and continuous work on Buildbot.

Best regards,
Jan

On 2020-09-21 21:31, Pierre Tardy wrote:
> Hi Jan,
> Indeed multimaster will probably help for your scalability issues.
> 
> For performance monitoring, what takes 5min?
> Normally the UI is divided into several api calls, so it might slowly build, or is there 5min wait before displaying anything?
> You might want to use https://pypi.org/project/buildbot-profiler/ in order to see what function in buildbot is using too much CPU.
> 
> Also, what can impact performance is some blocking call made inside custom steps.
> For example, if you are using the `requests` library for doing http calls, then this could impact a lot of the performance.
> Those should be replaced by equivalent async (treq/txrequest)
> 
> Regards
> Pierre
> 
> 
> Le ven. 18 sept. 2020 à 17:33, Jan Marker <jan.marker at kdab.com <mailto:jan.marker at kdab.com>> a écrit :
> 
>     Hello,
> 
>     I was wondering whether you have any experience/advise about a two master configuration, where one master is serving the web service and the other is coordinating builds as mentioned in the documentation [0].
> 
>     We're talking of a setup with about 500-1500 builds of about 140 builders per day.
>     The web interface can take up to a minute to respond, certain pages might take up to 5 minutes.
>     The system is using a MySQL database and presumably reasonable caching values, but we do see multi-minute delays from buildrequest arrival to build start and the aforementioned web interface performance.
> 
>     So were wondering whether that kind of multi-master setup might help us out and improve the performance metrics.
>     I'd appreciate any kind of advice or experience with multi-master setups.
> 
>     Thanks,
>     Jan Marker
> 
>     [0] http://docs.buildbot.net/2.8.4/manual/concepts.html#multimaster
> 
>     -- 
>     Jan Marker | jan.marker at kdab.com <mailto:jan.marker at kdab.com> | Software Engineer
>     KDAB (Deutschland) GmbH, a KDAB Group company
>     Tel: +49-30-521325470
>     KDAB - The Qt, C++ and OpenGL Experts
> 
>     _______________________________________________
>     users mailing list
>     users at buildbot.net <mailto:users at buildbot.net>
>     https://lists.buildbot.net/mailman/listinfo/users
> 

-- 
Jan Marker | jan.marker at kdab.com | Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4310 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.buildbot.net/pipermail/users/attachments/20201114/d63985cb/attachment.bin>