[users at bb.net] Experiences with two Master Setup 1st web 2nd builds?

[Jan Marker]

Hi Pierre,

Thank you very much for your response.

For performance measurement we've configured our Apache HTTP proxy to measure the Time To First Byte [0] of all API calls.
The changes API is particularly slow ( /api/v2/builds/<build id>/changes ) even after including the recent fix [1], with some calls taking up to 5 minutes until the first byte arrives at the proxy.
Also requesting builds via the force schedulers (every builder has one) takes on average around 40-50 seconds.

Thanks for your suggestions regarding the buildbot-profiler project and the blocking calls.
I'll check both out.

Best regards,
Jan


[0] https://httpd.apache.org/docs/2.4/mod/mod_logio.html
[1] https://github.com/buildbot/buildbot/pull/5505

On 2020-09-21 21:31, Pierre Tardy wrote:
> Hi Jan,
> Indeed multimaster will probably help for your scalability issues.
> 
> For performance monitoring, what takes 5min?
> Normally the UI is divided into several api calls, so it might slowly build, or is there 5min wait before displaying anything?
> You might want to use https://pypi.org/project/buildbot-profiler/ in order to see what function in buildbot is using too much CPU.
> 
> Also, what can impact performance is some blocking call made inside custom steps.
> For example, if you are using the `requests` library for doing http calls, then this could impact a lot of the performance.
> Those should be replaced by equivalent async (treq/txrequest)
> 
> Regards
> Pierre
> 
> 
> Le ven. 18 sept. 2020 à 17:33, Jan Marker <jan.marker at kdab.com <mailto:jan.marker at kdab.com>> a écrit :
> 
>     Hello,
> 
>     I was wondering whether you have any experience/advise about a two master configuration, where one master is serving the web service and the other is coordinating builds as mentioned in the documentation [0].
> 
>     We're talking of a setup with about 500-1500 builds of about 140 builders per day.
>     The web interface can take up to a minute to respond, certain pages might take up to 5 minutes.
>     The system is using a MySQL database and presumably reasonable caching values, but we do see multi-minute delays from buildrequest arrival to build start and the aforementioned web interface performance.
> 
>     So were wondering whether that kind of multi-master setup might help us out and improve the performance metrics.
>     I'd appreciate any kind of advice or experience with multi-master setups.
> 
>     Thanks,
>     Jan Marker
> 
>     [0] http://docs.buildbot.net/2.8.4/manual/concepts.html#multimaster
> 
>     -- 
>     Jan Marker | jan.marker at kdab.com <mailto:jan.marker at kdab.com> | Software Engineer
>     KDAB (Deutschland) GmbH, a KDAB Group company
>     Tel: +49-30-521325470
>     KDAB - The Qt, C++ and OpenGL Experts
> 
>     _______________________________________________
>     users mailing list
>     users at buildbot.net <mailto:users at buildbot.net>
>     https://lists.buildbot.net/mailman/listinfo/users
> 

-- 
Jan Marker | jan.marker at kdab.com | Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4310 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.buildbot.net/pipermail/users/attachments/20200927/3945cd96/attachment.bin>