[Buildbot-commits] [Buildbot] #2706: Handling test results

[Buildbot trac]

#2706: Handling test results
-------------------------+-------------------
Reporter:  jpommerening  |      Owner:
    Type:  enhancement   |     Status:  new
Priority:  minor         |  Milestone:  0.8.+
 Version:  0.8.8         |   Keywords:
-------------------------+-------------------
 Me again…

 I'm currently working on integrating our test results with the BuildBot
 web-frontend and mail notifications. I discovered the "halfway there"
 implementation consisting of `BuildStatus.addTestResult` some unfinished
 web views etc. and thought now would be a good opportunity to tackle that
 problem.

 I already whipped up a little proof-of-concept based on 0.8.8 and am now
 looking to clean up some rough edges and learn how make the right
 extensions to nine's the data api (?).

 '''First: What's there already?'''

 [https://github.com/buildbot/buildbot/blob/master/master/buildbot/interfaces.py#L595
 ITestResult]: Everything that is known about a single test case: Its name
 (a tuple of strings), result, a string/text further describing the result,
 a dictionary (string->string) of logs.

 [https://github.com/buildbot/buildbot/blob/master/master/buildbot/interfaces.py#L577
 IBuildStatus.getTestResults]: Returns all test results for the given
 build, keeping `ITestResult` per name (where name is
 `ITestResult.getName()`).

 [https://github.com/buildbot/buildbot/blob/master/master/buildbot/status/web/tests.py
 StatusResourceBuildTest]: a web view that can display the result and logs
 of a single test case.

 '''What do I want to contribute?'''

 1. `ILogObserver` implementations for incrementally parsing test result
 logs
    * SubunitLogObserver – already there
    * JUnitLogObserver – parse incrementally with defusedxml (I'll port my
 lxml-based implementation)
    * TAPLogObserver – can someone point me to a python library for that?
    * are there any other wide spread formats?

 2. I want to replace the dictionary with a list (possible upgrade hazard?)
    * currently the test results are in whatever order the dictionary deems
 right, the list would assure chronological order
    * Allows duplicate names, I'll have to see how to handle that in the
 ui. I actually think this is needed for data-driven tests (a single
 method, multiple datasets that are tested within the same test, multiple
 results).

 3. A "click-through/drill-down" web interface to navigate test results on
 0.8.x (#1794)
    * linked on the build-status page in the "Result" section with number
 of passed and failed tests (#536)
    * tests results grouped by common prefix of the name tuple (one page
 per "level")
    * maybe a "proportionally split bar" style visualization of failed vs.
 passed tests

 4. hook it up in nine
    * I have no idea what that means, can someone point me in the right
 direction?

 (each of these steps may come as a separate pull request on github)

 '''Why is this important?'''

 This may not yet be the final solution but I think it is a step in the
 right direction. Buildbot is all about CI and proper integration of test
 tools should be a big part of that. Right now the whole test result thing
 is kind of in stealth mode – I didn't know about it until last week,
 despite having worked with Buildbot for about 2 years. The changes I want
 to make would show that there '''is''' a feature like that and maybe
 inspire discussion and ideas for improving it.

 '''What feedback do I need?'''

 Can I go ahead and change `getTestResults` to return a list?
 What changes are needed for nine?
 Do you have any suggestions, things I should consider, etc?

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2706>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation