[Buildbot-devel] GSoC: Initial thoughts on the Graphs and Data Charts Project

[Mikhail Sobolev]

On Tue, Mar 17, 2015 at 12:14:46PM +0530, Prasoon Shukla wrote:
>    Well, I know the way I write might not be very clear. Anyway.
>    ᐧ
>    On Tue, Mar 17, 2015 at 11:15 AM, Mikhail Sobolev <mss at mawhrin.net>
>    wrote:
> 
>      On Tue, Mar 17, 2015 at 02:45:02AM +0530, Prasoon Shukla wrote:
>      Can you please provide examples of the data that are not inside Buildbot
>      and that you consider to provide graphs for?
> 
>     
>    As a solid example, I might want to measure the size of my generated
>    binaries, or to measure the number of function calls that a particular
>    unit-test makes, or to measure the total lines of comments in my code, or
>    to measure the how long a particular test runs for (and not the whole
>    build).
Excellent example.  So the question to ask (and implement as a part of this
project) is: how do I make Buildbot aware of this data and how to turn this
data into a metric?

>    Just to illustrate with an example, the test suite for LLVM is written
>    using [2]google test framework. Then, we cannot use metrics code to
>    measure/count stuff within the LLVM test suite. This is what I meant when
>    I said that metrics code can not be used outside buildbot.
>    So, for all non-trivial build data (e.g. timing a single unit test) we
>    cannot use metrics module.
I do not agree with the conclusion ("we cannot use metrics module"), however I
must note that this is another very good example for the point above.

>> It would be nice if you provided examples for both kinds, then it'd be a bit
>> easier follow your reasoning.

>    1. For time based test-statistic: Taking the example from above, say I
>    want to measure how long it takes a run a particularly time-consuming
>    test. I will make that one test run as a separate Step. So, buildbot will
>    measure the time taken for this Step automatically (since it measures
>    times of all steps). Then, we could have queried for this data over a
>    number of builds and plotted it. (But since we're planning on using
>    influxDB, this will not be done.)
>    2. For non-time based test-statistic: If I want to measure the size of my
>    final build (for example, the size of generated binaries), I'll make a
>    small script that measure the size of the build directory and output it to
>    stdout. Then, I'll have this script run from within a new Step that can
>    record this build size from stdout and store it in the db.
Very good examples.  Let's repeat the mantra: how do I turn this data into a
metric understood by Buildbot so it's ...

>    I will make sure to include examples from now on, wherever possible.
Please do!

-- 
Misha