[Buildbot-commits] [Buildbot] #2461: Add support for providing and graphing data charts of build statistics over time.

[Buildbot trac]

#2461: Add support for providing and graphing data charts of build statistics over
time.
-------------------------+--------------------
Reporter:  juj           |       Owner:
    Type:  project-idea  |      Status:  new
Priority:  major         |   Milestone:  0.9.+
 Version:  0.8.7p1       |  Resolution:
Keywords:                |
-------------------------+--------------------

Comment (by juj):

 I have recently added this feature into MathGeoLib, see here:

 http://clb.demon.fi:8113/waterfall
 http://clb.demon.fi/dump/MathGeoLib_testresults/index.html?revision=4ff29c9d3537071c9a8ad81e4425bceb2fccb7fc

 How does it work?

 1. Each build slave builds a test executable as part of its build.
 2. The test executable is programmed to output a test results JSON file to
 local disk. See an example:
 http://clb.demon.fi/dump/MathGeoLib_testresults/e8287b43e7e9d416f2d0771889d856212af8e7f3-vs2012
 -32bit-sse41.json
 3. A FileUpload step is performed to upload this to WWW server. The WWW
 server stores all .json result files across time in a single web-
 accessible folder.
 4. A custom WWW page was created
 (http://clb.demon.fi/dump/MathGeoLib_testresults/index.html) that
    a. is given a HTTP GET parameter of where to locate the JSON results
 files.
    b. loads all JSON files (by git revision hash) using jQuery.
    c. assembles the loaded JSON data into a format displayable using
 HighCharts.
    d. Outputs results graph to user.

 I see that anyone looking to integrate this to their own project will need
 to perform similar kind of custom results file creation process
 via their application, to be able to graph custom data.

 This whole process is very specific to MathGeoLib. I see a lot of room for
 further work:
   - Generalize the machinery to that it is useful for other domains, open
 questions:
      - Should the results JSON file format be 'standardized' on?
      - What other types of graphs would be useful?
   - Simplify the machinery that it's easier to integrate to new projects.
   - Integrate the graph rendering page to waterfall.
   - Add support for drawing graphs for changes over time. The current page
 draws data only for a single git commit hash.
   - Add support for drawing graphs from internal data produced by
 Buildbot. By internal data I mean e.g. the properties produced by build
 steps.
     Example: http://clb.demon.fi:8113/builders/vs2012-MathGeoLib-32bit-
 SSE4.1/builds/24
     One might want to graph time elapsed in the build step, and how it
 varies across time. Users can also defined properties, and there are
 properties specific to certain steps, like the "warnings-count" on the
 example page.

 There is a JSON API to buildbot to retrieve these properties, so perhaps
 that would be a starting point.

 On a very high-level scale, I see there being two types of data: build
 properties that are automatically produced by BuildBot steps, and custom
 data that's internal to the project and possibly very domain-specific,
 like the profiling data generated by MathGeoLib.

 Graphing buildbot-generated data should be super-easy (could be automatic
 even without user configuration needed?), but graphing project-generated
 data will definitely require a custom procedure like the one developed as
 a proof-of-concept above.

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2461#comment:4>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation