[Buildbot-devel] build log files

[Brian Warner]

"Bailey, Darragh" <dbailey at hp.com> writes:

> Noticed that my build logs take up a lot of space. Since I have 64
> builds running and the largest log is at 11MB, this can rapidly grow to
> take a lot of space. On average, this looks to be about 300MB per day.
>
> So I'm wondering if anyone has done any work on supporting compressed
> log files in buildbot. I would like to keep about 2 week's worth of log
> files. But not if it's going to take up 4GB of space.

Hm, that's a good idea. It isn't trivial to implement, since the logfiles are
stored in a slightly weird way (designed to allow incremental-append while
other people are reading from them, and to accomodate multiple "channels"
like stdout vs stderr), but it isn't an impossible task.

I've created (trac) ticket #26 for this issue.

> Would this be something worthwhile adding to buildbot in general or do I
> just have unusually large log files? 

I think is *would* be worthwhile to add. I've found that my own buildmasters
are consuming a lot of space with logfiles too. My usual approach is to use a
crontab to prune the logs after a few weeks (note that you can prune the logs
while leaving the rest of the build status in place). But I suspect that my
builds are shorter and run less frequently than yours.

cheers,
 -Brian