[Buildbot-devel] GSoC: Initial thoughts on the Graphs and Data Charts Project

[Prasoon Shukla]

Okay. So from the last couple of posts, Buildbot can already gather data
for the supported frameworks. This gathering happens in the Steps. In case
someone wants to add support for a new framework, they can simply add a new
Step that can process output from *that* framework.
As an example, if I wanted to add support for JSLint, I'd have to write a
new Step like the one written for PyLint
<http://docs.buildbot.net/latest/manual/cfg-buildsteps.html#pylint>.

This also means  that we can use a configuration based setup for influx
that Dustin wrote about with no problems.

Mikhail wrote:

> First you need to think about how data provided by use of those classes
> will
> end up in any external storage.
> Second, _you_ should think if there's anything missing in the module.


That's just it - I would prefer not to use the metrics module at all. Why?
Well, right now, data is being gathered inside a Step. Then, we can do one
of the two things:
1. Pass this data directly to influxDB (using the Python APi for influx
<https://github.com/influxdb/influxdb-python>), or,
2. Pass this data to the metrics module, log it, and *then* pass the data
to influx.

(Again, let me use an example. Suppose a Step has collected the number of
skipped tests. Then, Step can either directly call a function to store this
number in the influx databases via an API call. Or, the Step can call a
function in metrics module - in this case, MetricCountEvent.log('Tests
skipped', skipped_tests) - which will then call *another* function to store
this number in the influx database).

Then, there is simply no use for the option #2. This is because the data is
going to be stored in the influxDB in both cases and so, there is no point
of logging the data too. If I want to view the number of skipped tests, I
can simply use the influxDB interface for it. Or even better, I can
visualize this number on a skipped_tests vs build graph.

If the core devs *really* want me to use the metrics module as middleman,
please provide me the reasoning for it. I might be wrong, of course, and
there might be a good reason behind using the metrics module but I cannot
see any such reason at present.

>    2. How will we collect, say, the number of skipped tests?
> That's basically one of the points of this project: understand, design and
> implement a way for certain data provided by various components to appear
> as a
> metric, which is then collected and stored.


So, as was mentioned in the post before, buildbot is already capable of
collecting this data and so, this is a non issue. For other types of data
that buildbot cannot collect, I would like to go with the script based
approach I mentioned in an earlier email. Again, let's have an example.

So, say I want to measure the total number of function calls resulting from
a certain unit-test function. If our tests are in python, then we can use
the traceback module to measure this number. I will run the tests as usual
and this number will be measured. Then, I'll write this number to a
temporary file and make a simple script that echos this number to stdout.
This script I can put in a new Step which will read this number as usual
and pass it off to influx to be stored.

This approach takes care of getting buildbot to gather arbitrary data.

If you think something is "hard", please provide rationale; that will help
> others to follow your reasoning and provide additional information and/or a
> different point of view should it be necessary.


I thought it as hard precisely because I expected that we'd have to write a
truckload of test-output parsers. Now that I know that buildbot already
supports a lot of data collection within Steps and that we can ignore the
unsupported frameworks, the task doesn't seem hard.

>    Just to illustrate with an example, the test suite for LLVM is written
> >    using [2]google test framework. Then, we cannot use metrics code to
> >    measure/count stuff within the LLVM test suite. This is what I meant
> when
> >    I said that metrics code can not be used outside buildbot.
> >    So, for all non-trivial build data (e.g. timing a single unit test) we
> >    cannot use metrics module.
> I do not agree with the conclusion ("we cannot use metrics module"),
> however I
> must note that this is another very good example for the point above.


Well, please provide a counter-example in that case. How would you use the
buildbot.metrics code inside a C project? Would you write a Python-C
interface to interact with the buildbot code? Even if you do that, it would
couple two independent system - I would, for example, need buildbot to run
the LLVM test suite manually. This would be an absurd requirement for a
normal dev who isn't interacting with buildbot and sees it as a blackbox.

how do I make Buildbot aware of this data and how to turn this
> data into a metric


You keep using that phrase. Unfortunately, I don't quite understand what
you mean by *turning* the build data into metric. Please explain this by
way of an example. Thank you.

Prasoon
ᐧ

On Tue, Mar 17, 2015 at 11:59 PM, Mikhail Sobolev <mss at mawhrin.net> wrote:

> On Tue, Mar 17, 2015 at 12:14:46PM +0530, Prasoon Shukla wrote:
> >    Well, I know the way I write might not be very clear. Anyway.
> >    ᐧ
> >    On Tue, Mar 17, 2015 at 11:15 AM, Mikhail Sobolev <mss at mawhrin.net>
> >    wrote:
> >
> >      On Tue, Mar 17, 2015 at 02:45:02AM +0530, Prasoon Shukla wrote:
> >      Can you please provide examples of the data that are not inside
> Buildbot
> >      and that you consider to provide graphs for?
> >
> >
> >    As a solid example, I might want to measure the size of my generated
> >    binaries, or to measure the number of function calls that a particular
> >    unit-test makes, or to measure the total lines of comments in my
> code, or
> >    to measure the how long a particular test runs for (and not the whole
> >    build).
> Excellent example.  So the question to ask (and implement as a part of this
> project) is: how do I make Buildbot aware of this data and how to turn this
> data into a metric?
>
> >    Just to illustrate with an example, the test suite for LLVM is written
> >    using [2]google test framework. Then, we cannot use metrics code to
> >    measure/count stuff within the LLVM test suite. This is what I meant
> when
> >    I said that metrics code can not be used outside buildbot.
> >    So, for all non-trivial build data (e.g. timing a single unit test) we
> >    cannot use metrics module.
> I do not agree with the conclusion ("we cannot use metrics module"),
> however I
> must note that this is another very good example for the point above.
>
> >> It would be nice if you provided examples for both kinds, then it'd be
> a bit
> >> easier follow your reasoning.
>
> >    1. For time based test-statistic: Taking the example from above, say I
> >    want to measure how long it takes a run a particularly time-consuming
> >    test. I will make that one test run as a separate Step. So, buildbot
> will
> >    measure the time taken for this Step automatically (since it measures
> >    times of all steps). Then, we could have queried for this data over a
> >    number of builds and plotted it. (But since we're planning on using
> >    influxDB, this will not be done.)
> >    2. For non-time based test-statistic: If I want to measure the size
> of my
> >    final build (for example, the size of generated binaries), I'll make a
> >    small script that measure the size of the build directory and output
> it to
> >    stdout. Then, I'll have this script run from within a new Step that
> can
> >    record this build size from stdout and store it in the db.
> Very good examples.  Let's repeat the mantra: how do I turn this data into
> a
> metric understood by Buildbot so it's ...
>
> >    I will make sure to include examples from now on, wherever possible.
> Please do!
>
> --
> Misha
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20150318/f29cea00/attachment.html>