[Buildbot-devel] Storing compile/test results in a database?

[Axel Hecht]

I have something like this running (and the stuff that Markus talked about).
catlee is experimenting with a build database, too.

I'm not doing any async SQL work. Mostly because I don't speak SQL, and
there are no twisted-based SQL abstractions. At least none worthwhile
mentioning.

Over to real life:

For the detailed data reports, I'm having a custom step, which is parsing
the result of the command and feeds it into a db. This may or may not be an
implementation detail for you, I'm actually having a custom slave command,
too, which is feeding structured data into a separate channel in the log as
json, and I'm using that data to feed my db. That way, i don't have to
create cumbersome machine-readable information on stdout. This all ends up
in a remoteUpdate method in a LoggedRemoteCommand subclass,
http://hg.mozilla.org/users/axel_mozilla.com/locale-inspector/file/20d43d071cf0/l10ninsp/steps.py#l70
.

This does imply that your db is set up such that doing sync writes is OK. We
don't have too much real life experience with that, but if you'd get to the
point that it's not, adding slave dbs for the query work sounds like a more
standard solution than adding a fancy twisted db abstraction and having to
figure out what the concurrency model should be. Which is what the twisted
folks seem to have argued about years ago from what I found on google.

Axel

2009/6/9 Kristian Nielsen <knielsen at knielsen-hq.org>

> I know that it has already been discussed to use an SQL database for
> backend
> storage of Buildbot state and history, and that the idea is for this to go
> in
> 1.0. However, I have a need to get some data in a database now, and my
> needs
> are also somewhat different, so I wanted to bounce off some ideas and hear
> if
> there is already some existing effort that I could build on.
>
> We are using Buildbot for automatic testing of MariaDB, a comunity branch
> of
> MySQL. The main thing here is to run the test suite. There are on the order
> of
> 1000 individual test cases run over a number of different branches. Just
> maintaining the test suite is a major effort, and we need a tool to help
> with
> this.
>
> Basically I want to store every test case failure in a database along with
> information about branch, revision, error output, etc. This will allow to
> easily create additional webpages, a facility to search for previous
> failures
> across different builds, etc. (At MySQL AB I wrote an internal tool for
> this,
> and the database was > 1 GB in size).
>
> My idea is basically to put in custom LogObserver hooks and maybe subclass
> build steps as necessary to get access to the information, just adding code
> to
> do the appropriate SQL INSERT statements. The actual code for this will be
> fairly specific to the way the MySQL test suite works.
>
> But I would like to hear if someone else already did something like this?
> Or
> if there already exists some generic support for this in Buildbot, or
> interest
> in using such generic support if I were to implement it?
>
> Eg. some mechanism to access the database in a good way to use connections
> in
> an efficient way and avoid blocking operations sounds useful. And maybe
> some
> mechanism to make all the needed information (revision, build number,
> branch,
> ...) easily available across the needed log observers.
>
> Any thoughts?
>
> Thanks,
>
>  - Kristian.
>
>
> ------------------------------------------------------------------------------
> Crystal Reports - New Free Runtime and 30 Day Trial
> Check out the new simplified licensing option that enables unlimited
> royalty-free distribution of the report engine for externally facing
> server and web deployment.
> http://p.sf.net/sfu/businessobjects
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20090609/5c4473aa/attachment.html>