[Buildbot-devel] Storing compile/test results in a database?

[Kristian Nielsen]

I know that it has already been discussed to use an SQL database for backend
storage of Buildbot state and history, and that the idea is for this to go in
1.0. However, I have a need to get some data in a database now, and my needs
are also somewhat different, so I wanted to bounce off some ideas and hear if
there is already some existing effort that I could build on.

We are using Buildbot for automatic testing of MariaDB, a comunity branch of
MySQL. The main thing here is to run the test suite. There are on the order of
1000 individual test cases run over a number of different branches. Just
maintaining the test suite is a major effort, and we need a tool to help with
this.

Basically I want to store every test case failure in a database along with
information about branch, revision, error output, etc. This will allow to
easily create additional webpages, a facility to search for previous failures
across different builds, etc. (At MySQL AB I wrote an internal tool for this,
and the database was > 1 GB in size).

My idea is basically to put in custom LogObserver hooks and maybe subclass
build steps as necessary to get access to the information, just adding code to
do the appropriate SQL INSERT statements. The actual code for this will be
fairly specific to the way the MySQL test suite works.

But I would like to hear if someone else already did something like this? Or
if there already exists some generic support for this in Buildbot, or interest
in using such generic support if I were to implement it?

Eg. some mechanism to access the database in a good way to use connections in
an efficient way and avoid blocking operations sounds useful. And maybe some
mechanism to make all the needed information (revision, build number, branch,
...) easily available across the needed log observers.

Any thoughts?

Thanks,

 - Kristian.