<div dir="ltr">Hi Finn,<div><br></div><div>I did start a while ago a scalability study on Buildbot nine, which I did not have time to completely finish yet.</div><div><br></div><div>When looking at the profiles of Buildbot under load, we can see that the most important load contributor is the compression and storage of the logs in the sql db</div><div><a href="https://github.com/tardyp/buildbot_profiler">https://github.com/tardyp/buildbot_profiler</a><br></div><div><br></div><div>In order to assess how much the system can keep up with, I started up to 300 workers, and a variable number of masters.</div><div>The builders are configured to output 40000 lines of log as fast as possible.</div><div>I used an experimental marathon setup that we have at work, and I actually ran into stability issues of our nixy loadbalancing system, while running those tests.</div><div>But Buildbot did run flawlessly in this setup.</div><div><br></div><div>One the maximum load, I saw the postgresql running at max 80%cpu, while 4 masters were at full cpu load.</div><div><br></div><div>During that test, the whole system was able to eat 50000 lines of logs per seconds (20MB/s). As a log processing company, this might sound a bit few, but this is our current state.</div><div>I wasn't able to run with more masters and workers yet, as the marathon setup I used for this test wasn't big enough.</div><div>I really need to publish the numbers officially, though. But this will take a bit more time, as I want to also capture the cpu and network throughput stats during the test.</div><div><br></div><div>From the theorical point of view:</div><div>- buildbot is horizontally scalable, and you can create as much master as you want</div><div>- buildbot requires a sql database to store the data, this is for me the major brake in term of horizontal scalability. I did not manage yet to fully load postgresql though.</div><div>- there is also crossbar as a single point of failure, but given the number of messages that buildbot generates, and the perf numbers that crossbar publishes, I dont think it will be a proble,</div><div>  </div><div><br></div><div>I know that there are some other people on that list that run multimaster on prod, and which can give more inputs.</div><div>From what I can tell of the feedbacks he has been given here, the multimaster process was not very much the issue he had. I can remember a lot of discussion with live reconfiguration of schedulers, which we finally fixed in 0.9.4.</div><div><br></div><div>We also have the usage statistics that we collect since 0.9.2.</div><div><br></div><div>Here is the number master running on buildbot, sorting by number of builders. (for people who publish their stats)</div><div><div><table class="inbox-inbox-table inbox-inbox-table-condensed" style="box-sizing:border-box;border-collapse:collapse;width:332px;max-width:100%;margin-bottom:18px;color:rgb(68,68,68);font-family:lato,"helvetica neue",helvetica,arial,sans-serif"><thead style="box-sizing:border-box"><tr style="box-sizing:border-box"><th class="inbox-inbox-" style="box-sizing:border-box;padding:5px;text-align:left;line-height:1.42857;vertical-align:bottom;border-top:0px;border-bottom:2px solid rgb(236,240,241)"><span class="inbox-inbox-ng-binding" style="box-sizing:border-box">plugins.buildbot/config/BuilderConfig: Descending</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-fa inbox-inbox-ng-scope inbox-inbox-fa-sort-desc" style="box-sizing:border-box;display:inline-block;font-weight:normal;font-stretch:normal;line-height:1.42857;font-family:fontawesome;font-size:inherit;color:rgb(180,188,194)"></span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-fa inbox-inbox-fa-search inbox-inbox-ng-scope" style="box-sizing:border-box;display:inline-block;font-weight:normal;font-stretch:normal;line-height:1.42857;font-family:fontawesome;font-size:inherit"></span></th><th class="inbox-inbox-visualize-table-right" style="box-sizing:border-box;padding:5px;text-align:left;line-height:1.42857;vertical-align:bottom;border-top:0px;border-bottom:2px solid rgb(236,240,241)"><span class="inbox-inbox-ng-binding" style="box-sizing:border-box">Unique count of installid.raw</span><span class="inbox-inbox-fa inbox-inbox-ng-scope inbox-inbox-fa-sort" style="box-sizing:border-box;display:inline-block;font-weight:normal;font-stretch:normal;line-height:1.42857;font-family:fontawesome;font-size:inherit;color:rgb(180,188,194)"></span></th></tr></thead><tbody style="box-sizing:border-box"><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">114</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">3</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">102</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">2</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">98</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">1</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">94</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">13</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">93</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">1</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">90</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">1</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">84</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">9</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">82</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">21</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">81</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">1</td></tr></tbody></table>And here the number of master installation sorted by number of worker attached, again this is for the people that publish. This is why you don't see my tests here.<br><table class="inbox-inbox-table inbox-inbox-table-condensed" style="box-sizing:border-box;border-collapse:collapse;width:331px;max-width:100%;margin-bottom:18px;color:rgb(68,68,68);font-family:lato,"helvetica neue",helvetica,arial,sans-serif"><thead style="box-sizing:border-box"><tr style="box-sizing:border-box"><th class="inbox-inbox-" style="box-sizing:border-box;padding:5px;text-align:left;line-height:1.42857;vertical-align:bottom;border-top:0px;border-bottom:2px solid rgb(236,240,241)"><span class="inbox-inbox-ng-binding" style="box-sizing:border-box">plugins.buildbot/worker/base/Worker: Descending</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-fa inbox-inbox-ng-scope inbox-inbox-fa-sort-desc" style="box-sizing:border-box;display:inline-block;font-weight:normal;font-stretch:normal;line-height:1.42857;font-family:fontawesome;font-size:inherit;color:rgb(180,188,194)"></span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-fa inbox-inbox-fa-search inbox-inbox-ng-scope" style="box-sizing:border-box;display:inline-block;font-weight:normal;font-stretch:normal;line-height:1.42857;font-family:fontawesome;font-size:inherit"></span></th><th class="inbox-inbox-visualize-table-right" style="box-sizing:border-box;padding:5px;text-align:left;line-height:1.42857;vertical-align:bottom;border-top:0px;border-bottom:2px solid rgb(236,240,241)"><span class="inbox-inbox-ng-binding" style="box-sizing:border-box">Unique count of installid.raw</span><span class="inbox-inbox-fa inbox-inbox-ng-scope inbox-inbox-fa-sort" style="box-sizing:border-box;display:inline-block;font-weight:normal;font-stretch:normal;line-height:1.42857;font-family:fontawesome;font-size:inherit;color:rgb(180,188,194)"></span></th></tr></thead><tbody style="box-sizing:border-box"><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">52</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">5</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">49</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">5</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">46</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">13</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">27</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">35</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">26</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">15</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">22</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">5</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">14</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">17</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">13</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">24</td></tr><tr style="box-sizing:border-box"><td class="inbox-inbox-cell-hover inbox-inbox-ng-scope" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">10</td><td class="inbox-inbox-numeric-value" style="box-sizing:border-box;padding:5px;line-height:1.42857;vertical-align:top;border-top:1px solid rgb(236,240,241)">6</td></tr></tbody></table>We don't upload data on multi-master, so I can't tell how many multimaster instance we have.</div></div><div><br></div><div><br></div><div>I would love to see more people using Buildbot at large scale, and help fixing the performance issues.</div><div><br></div><div>Regards,</div><div>Pierre</div></div><br><div class="gmail_quote"><div dir="ltr">Le jeu. 23 mars 2017 à 19:26, Finn Herzfeld <<a href="mailto:fherzfeld@splunk.com">fherzfeld@splunk.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">







<div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72" class="gmail_msg">
<div class="m_-5787289896999278871WordSection1 gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">Hey all,<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">We’re evaluating using Buildbot here at Splunk, we think it’s really cool and could potentially solve a lot of our problems. One of the questions that’s come up is how large it can scale in a multi-master
 environment. How large have others scaled it? What resources are going to be constrained when scaling it? What issues are people running into with the current state of multi-master? I see that it’s is considered experimental.<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">Thanks,<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">Finn Herzfeld<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
</div>
</div>

_______________________________________________<br class="gmail_msg">
users mailing list<br class="gmail_msg">
<a href="mailto:users@buildbot.net" class="gmail_msg" target="_blank">users@buildbot.net</a><br class="gmail_msg">
<a href="https://lists.buildbot.net/mailman/listinfo/users" rel="noreferrer" class="gmail_msg" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a></blockquote></div>