<div dir="ltr">Hello all,<div><br></div><div>It has always been very difficult for us to understand how buildbot is used, and how much people are upgrading to newer version</div><div><br></div><div>There is of course <a href="http://trac.buildbot.net/wiki/SuccessStories">http://trac.buildbot.net/wiki/SuccessStories</a> but beyond that there are tons of small shops using buildbot quietly in a variety of environment.</div><div><br></div><div>Having a good idea of how much it is used will also help people to embrace buildbot nine even more, knowing how much other people already use it without problems.</div><div><br></div><div>This is why we will introduce the buildbotNetUsageData feature in buildbot 0.9.0rc3.</div><div>The idea of this feature is to send usage data to our <a href="http://buildbot.net">buildbot.net</a> infrastructure.</div><div>In the tradition of buildbot, this feature is very configurable, so that you can disable it or filter the data sent as you want.</div><div><br></div><div><a href="https://github.com/buildbot/buildbot/pull/2393">https://github.com/buildbot/buildbot/pull/2393</a><br></div><div>We wanted to heads-up the mailing list before this lands</div><div><br></div><div><br></div><div> Following is a copy of the documentation for your reference:</div><div><br></div><div><p style="font-family:"open sans",sans-serif;font-size:15px;line-height:21px;background-color:rgb(246,246,246)">Since buildbot 0.9.0, buildbot has a simple feature which sends usage analysis info to <a href="http://buildbot.net">buildbot.net</a>. This is very important for buildbot developers to understand how the community is using the tools. This allows to better prioritize issues, and understand what plugins are actually being used. This will also be a tool to decide whether to keep support for very old tools. For example buildbot contains support for the venerable CVS, but we have no information whether it actually works beyond the unit tests. We rely on the community to test and report issues with the old features.</p><p style="font-family:"open sans",sans-serif;font-size:15px;line-height:21px;background-color:rgb(246,246,246)">With BuildbotNetUsageData, we can know exactly what combination of plugins are working together, how much people are customizing plugins, what versions of the main dependencies people run.</p><p style="font-family:"open sans",sans-serif;font-size:15px;line-height:21px;background-color:rgb(246,246,246)">We take your privacy very seriously.</p><p style="font-family:"open sans",sans-serif;font-size:15px;line-height:21px;background-color:rgb(246,246,246)">BuildbotNetUsageData will never send information specific to your Code or Intellectual Property. No repository url, shell command values, host names, ip address or custom class names. If it does, then this is a bug, please report.</p><p style="font-family:"open sans",sans-serif;font-size:15px;line-height:21px;background-color:rgb(246,246,246)">We still need to track unique number for installation. This is done via doing a sha1 hash of master's hostname, installation path and fqdn. Using a secure hash means there is no way of knowing hostname, path and fqdn given the hash, but still there is a different hash for each master.</p><p style="font-family:"open sans",sans-serif;font-size:15px;line-height:21px;background-color:rgb(246,246,246)">You can see exactly what is sent in the master's twisted.log. Usage data is sent everytime the master is started.</p><p style="font-family:"open sans",sans-serif;font-size:15px;line-height:21px;background-color:rgb(246,246,246)">BuildbotNetUsageData can be configured with 4 values:</p><ul style="font-family:"open sans",sans-serif;font-size:15px;line-height:21px;background-color:rgb(246,246,246)"><li><p class="inbox-inbox-first" style="margin-top:0px"><code class="inbox-inbox-docutils inbox-inbox-literal"><span class="inbox-inbox-pre">c['buildbotNetUsageData']</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-pre">=</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-pre">None</span></code><span class="inbox-inbox-Apple-converted-space"> </span>disables the feature</p></li><li><p class="inbox-inbox-first" style="margin-top:0px"><code class="inbox-inbox-docutils inbox-inbox-literal"><span class="inbox-inbox-pre">c['buildbotNetUsageData']</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-pre">=</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-pre">'basic'</span></code><span class="inbox-inbox-Apple-converted-space"> </span>sends the basic information to buildbot including:</p><blockquote><div><ul class="inbox-inbox-simple"><li>versions of buildbot, python and twisted</li><li>platform information (CPU, OS, distribution, python flavor (i.e CPython vs PyPy))</li><li>mq and database type (mysql or sqlite?)</li><li>www plugins usage</li><li>Plugins usages: This counts the number of time each class of buildbot is used in your configuration. This counts workers, builders, steps, schedulers, change sources. If the plugin is subclassed, then it will be prefixed with a<span class="inbox-inbox-Apple-converted-space"> </span><cite>></cite></li></ul><p>example of basic report (for the metabuildbot):</p><div class="inbox-inbox-highlight-javascript"><div class="inbox-inbox-highlight" style="border-width:1px 1px 1px 3px;border-style:solid;border-color:rgb(204,204,204);font-size:1.1em;padding:10px;background-image:initial;background-color:rgb(255,255,255);background-size:initial;background-origin:initial;background-clip:initial;background-position:initial;background-repeat:initial"><pre style="overflow-x:auto;overflow-y:hidden;margin-top:0px;margin-bottom:0px"><span class="inbox-inbox-p">{</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'versions'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-p">{</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'Python'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'2.7.6'</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'Twisted'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'15.5.0'</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'Buildbot'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'0.9.0rc2-176-g5fa9dbf'</span>
<span class="inbox-inbox-p">},</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'platform'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-p">{</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'machine'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'x86_64'</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'python_implementation'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'CPython'</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'version'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'#140-Ubuntu SMP Mon Jul'</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'processor'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'x86_64'</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'distro:'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-p">(</span><span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'Ubuntu'</span><span class="inbox-inbox-p">,</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'14.04'</span><span class="inbox-inbox-p">,</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'trusty'</span><span class="inbox-inbox-p">)</span>
<span class="inbox-inbox-p">},</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'db'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'sqlite'</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'mq'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'simple'</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'plugins'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-p">{</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'buildbot.schedulers.forcesched.ForceScheduler'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-mi" style="color:rgb(0,153,153)">2</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'buildbot.schedulers.triggerable.Triggerable'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-mi" style="color:rgb(0,153,153)">1</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'buildbot.config.BuilderConfig'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-mi" style="color:rgb(0,153,153)">4</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'buildbot.schedulers.basic.AnyBranchScheduler'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-mi" style="color:rgb(0,153,153)">2</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'buildbot.steps.source.git.Git'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-mi" style="color:rgb(0,153,153)">4</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'>>buildbot.steps.trigger.Trigger'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-mi" style="color:rgb(0,153,153)">2</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'>>>buildbot.worker.base.Worker'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-mi" style="color:rgb(0,153,153)">4</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'buildbot.reporters.irc.IRC'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-mi" style="color:rgb(0,153,153)">1</span><span class="inbox-inbox-p">,</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'>>>buildbot.process.buildstep.LoggingBuildStep'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-mi" style="color:rgb(0,153,153)">2</span><span class="inbox-inbox-p">},</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'www_plugins'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-p">[</span><span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'buildbot_travis'</span><span class="inbox-inbox-p">,</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'waterfall_view'</span><span class="inbox-inbox-p">]</span>
<span class="inbox-inbox-p">}</span>
</pre></div></div></div></blockquote></li><li><p class="inbox-inbox-first" style="margin-top:0px"><code class="inbox-inbox-docutils inbox-inbox-literal"><span class="inbox-inbox-pre">c['buildbotNetUsageData']</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-pre">=</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-pre">'full'</span></code><span class="inbox-inbox-Apple-converted-space"> </span>sends the basic information plus additional information:</p><blockquote><div><ul class="inbox-inbox-simple"><li>configuration of each builders: how the steps are arranged together. for ex:</li></ul><div class="inbox-inbox-highlight-javascript"><div class="inbox-inbox-highlight" style="border-width:1px 1px 1px 3px;border-style:solid;border-color:rgb(204,204,204);font-size:1.1em;padding:10px;background-image:initial;background-color:rgb(255,255,255);background-size:initial;background-origin:initial;background-clip:initial;background-position:initial;background-repeat:initial"><pre style="overflow-x:auto;overflow-y:hidden;margin-top:0px;margin-bottom:0px"><span class="inbox-inbox-p">{</span>
<span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'builders'</span><span class="inbox-inbox-o" style="font-weight:bold">:</span> <span class="inbox-inbox-p">[</span>
<span class="inbox-inbox-p">[</span><span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'buildbot.steps.source.git.Git'</span><span class="inbox-inbox-p">,</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'>>>buildbot.process.buildstep.LoggingBuildStep'</span><span class="inbox-inbox-p">],</span>
<span class="inbox-inbox-p">[</span><span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'buildbot.steps.source.git.Git'</span><span class="inbox-inbox-p">,</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'>>buildbot.steps.trigger.Trigger'</span><span class="inbox-inbox-p">],</span>
<span class="inbox-inbox-p">[</span><span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'buildbot.steps.source.git.Git'</span><span class="inbox-inbox-p">,</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'>>>buildbot.process.buildstep.LoggingBuildStep'</span><span class="inbox-inbox-p">],</span>
<span class="inbox-inbox-p">[</span><span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'buildbot.steps.source.git.Git'</span><span class="inbox-inbox-p">,</span> <span class="inbox-inbox-s1" style="color:rgb(187,136,68)">'>>buildbot.steps.trigger.Trigger'</span><span class="inbox-inbox-p">]]</span>
<span class="inbox-inbox-p">}</span>
</pre></div></div></div></blockquote></li><li><p class="inbox-inbox-first" style="margin-top:0px"><code class="inbox-inbox-docutils inbox-inbox-literal"><span class="inbox-inbox-pre">c['buildbotNetUsageData']</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-pre">=</span><span class="inbox-inbox-Apple-converted-space"> </span><span class="inbox-inbox-pre">myCustomFunction</span></code>. You can also specify exactly what to send using a callback.</p><blockquote><div><p>The custom function will take the generated data from full report in the form of a dictionary, and return a customized report as a jsonable dictionary. You can use this to filter any information you dont want to disclose. You can use a custom http_proxy environment variable in order to not send any data while developing your callback.</p></div></blockquote></li></ul></div><div><br></div></div>