[Buildbot-commits] [Buildbot] #2706: Handling test results

[Buildbot trac]

#2706: Handling test results
-------------------------+--------------------
Reporter:  jpommerening  |       Owner:
    Type:  enhancement   |      Status:  new
Priority:  minor         |   Milestone:  0.8.+
 Version:  0.8.8         |  Resolution:
Keywords:                |
-------------------------+--------------------

Comment (by jpommerening):

 Some good points!


 > If you have thousands of builds (as most users do)
 check.
 > and each build has thousands of tests (as most test runs do),
 check…
 > that's millions of test-result rows.
 ouch… so what to do about it? If anything is good at handling that amount
 of equally structured data it should be a database, right?


 > It's a lot more for a larger-than-average installation. If we're sending
 a message for each new test result, that's also a lot of messages.
 > We've done some work with log handling in nine to try to reduce this (by
 collapsing multiple log lines into a single database record, and
 compressing it).

 This, indeed, sounds problematic. But wait, you're storing the logs in the
 database? And you were doing it line-by-line before? Also messages – is
 that related to nine's MQ stuff?

 My impression was that it worked something like this:

 * Slave watches logfile
 * Slave sends chunks to the master
 * LogObservers get updated with received chunks
 * Master pipes chunks to a file
 * Master compresses logfile, when step is done
 * file is read from disk and decompressed when it's needed

 But before I'll bother you with any more misinformed assumptions I'll have
 a look at the nine docs ;)

-- 
Ticket URL: <http://trac.buildbot.net/ticket/2706#comment:2>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation