<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.inbox-inbox-ng-binding
{mso-style-name:inbox-inbox-ng-binding;}
span.inbox-inbox-apple-converted-space
{mso-style-name:inbox-inbox-apple-converted-space;}
span.inbox-inbox-fa
{mso-style-name:inbox-inbox-fa;}
span.gmailmsg
{mso-style-name:gmail_msg;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Thanks, this is exactly the sort of numbers I was looking for. Very much appriciated<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">Pierre Tardy <tardyp@gmail.com><br>
<b>Date: </b>Thursday, March 23, 2017 at 2:01 PM<br>
<b>To: </b>Finn Herzfeld <fherzfeld@splunk.com>, "users@buildbot.net" <users@buildbot.net><br>
<b>Subject: </b>Re: [users@bb.net] Buildbot scaling limits<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hi Finn, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I did start a while ago a scalability study on Buildbot nine, which I did not have time to completely finish yet.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">When looking at the profiles of Buildbot under load, we can see that the most important load contributor is the compression and storage of the logs in the sql db<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://github.com/tardyp/buildbot_profiler">https://github.com/tardyp/buildbot_profiler</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In order to assess how much the system can keep up with, I started up to 300 workers, and a variable number of masters.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">The builders are configured to output 40000 lines of log as fast as possible.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I used an experimental marathon setup that we have at work, and I actually ran into stability issues of our nixy loadbalancing system, while running those tests.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">But Buildbot did run flawlessly in this setup.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">One the maximum load, I saw the postgresql running at max 80%cpu, while 4 masters were at full cpu load.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">During that test, the whole system was able to eat 50000 lines of logs per seconds (20MB/s). As a log processing company, this might sound a bit few, but this is our current state.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I wasn't able to run with more masters and workers yet, as the marathon setup I used for this test wasn't big enough.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I really need to publish the numbers officially, though. But this will take a bit more time, as I want to also capture the cpu and network throughput stats during the test.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">From the theorical point of view:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- buildbot is horizontally scalable, and you can create as much master as you want<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- buildbot requires a sql database to store the data, this is for me the major brake in term of horizontal scalability. I did not manage yet to fully load postgresql though.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- there is also crossbar as a single point of failure, but given the number of messages that buildbot generates, and the perf numbers that crossbar publishes, I dont think it will be a proble,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I know that there are some other people on that list that run multimaster on prod, and which can give more inputs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">From what I can tell of the feedbacks he has been given here, the multimaster process was not very much the issue he had. I can remember a lot of discussion with live reconfiguration of schedulers, which we finally fixed in 0.9.4.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We also have the usage statistics that we collect since 0.9.2.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Here is the number master running on buildbot, sorting by number of builders. (for people who publish their stats)<o:p></o:p></p>
</div>
<div>
<div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="332" style="width:249.0pt;border-collapse:collapse;box-sizing:border-box;max-width:100%">
<thead>
<tr style="box-sizing:border-box">
<td valign="bottom" style="border:none;border-bottom:solid #ECF0F1 1.5pt;padding:3.75pt 3.75pt 3.75pt 3.75pt">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span class="inbox-inbox-ng-binding"><b><span style="font-family:Helvetica;color:#444444">plugins.buildbot/config/BuilderConfig: Descending</span></b></span><span class="inbox-inbox-apple-converted-space"><b><span style="font-family:Helvetica;color:#444444"> </span></b></span><b><span style="font-family:Helvetica;color:#444444"><o:p></o:p></span></b></p>
</td>
<td valign="bottom" style="border:none;border-bottom:solid #ECF0F1 1.5pt;padding:3.75pt 3.75pt 3.75pt 3.75pt">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span class="inbox-inbox-ng-binding"><b><span style="font-family:Helvetica;color:#444444">Unique count of installid.raw</span></b></span><b><span style="font-family:Helvetica;color:#444444"><o:p></o:p></span></b></p>
</td>
</tr>
</thead>
<tbody>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">114<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">3<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">102<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">2<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">98<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">1<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">94<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">13<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">93<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">1<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">90<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">1<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">84<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">9<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">82<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">21<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">81<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">1<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal">And here the number of master installation sorted by number of worker attached, again this is for the people that publish. This is why you don't see my tests here.<o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="331" style="width:248.25pt;border-collapse:collapse;box-sizing:border-box;max-width:100%">
<thead>
<tr style="box-sizing:border-box">
<td valign="bottom" style="border:none;border-bottom:solid #ECF0F1 1.5pt;padding:3.75pt 3.75pt 3.75pt 3.75pt">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span class="inbox-inbox-ng-binding"><b><span style="font-family:Helvetica;color:#444444">plugins.buildbot/worker/base/Worker: Descending</span></b></span><span class="inbox-inbox-apple-converted-space"><b><span style="font-family:Helvetica;color:#444444"> </span></b></span><b><span style="font-family:Helvetica;color:#444444"><o:p></o:p></span></b></p>
</td>
<td valign="bottom" style="border:none;border-bottom:solid #ECF0F1 1.5pt;padding:3.75pt 3.75pt 3.75pt 3.75pt">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span class="inbox-inbox-ng-binding"><b><span style="font-family:Helvetica;color:#444444">Unique count of installid.raw</span></b></span><b><span style="font-family:Helvetica;color:#444444"><o:p></o:p></span></b></p>
</td>
</tr>
</thead>
<tbody>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">52<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">5<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">49<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">5<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">46<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">13<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">27<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">35<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">26<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">15<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">22<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">5<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">14<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">17<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">13<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">24<o:p></o:p></span></p>
</td>
</tr>
<tr style="box-sizing:border-box">
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">10<o:p></o:p></span></p>
</td>
<td valign="top" style="border:none;border-top:solid #ECF0F1 1.0pt;padding:3.75pt 3.75pt 3.75pt 3.75pt;box-sizing:border-box">
<p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-family:Helvetica;color:#444444">6<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal">We don't upload data on multi-master, so I can't tell how many multimaster instance we have.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I would love to see more people using Buildbot at large scale, and help fixing the performance issues.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Pierre<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Le jeu. 23 mars 2017 à 19:26, Finn Herzfeld <<a href="mailto:fherzfeld@splunk.com">fherzfeld@splunk.com</a>> a écrit :<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt">Hey all,</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt">We’re evaluating using Buildbot here at Splunk, we think it’s really cool and could potentially solve a lot of our problems.
One of the questions that’s come up is how large it can scale in a multi-master environment. How large have others scaled it? What resources are going to be constrained when scaling it? What issues are people running into with the current state of multi-master?
I see that it’s is considered experimental.</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt">Thanks,</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt">Finn Herzfeld</span></span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@buildbot.net" target="_blank">users@buildbot.net</a><br>
<a href="https://lists.buildbot.net/mailman/listinfo/users" target="_blank">https://lists.buildbot.net/mailman/listinfo/users</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>