[Buildbot-devel] build log files

[Bailey, Darragh]

I was originally thinking of on compressing the files when finished
writing to them. 

My thoughts were that it should be possible to compress the file when
finished writing to it, using the standard suffix of gz to indicate it
is compressed with gzip, then delete the original. 

I figured that you could enable this behaviour without impacting
backwards compatibility. And you can switch back at a later stage as
well. Also I would expect that reasonable amount of the read access to
the log files occur when they are being written, so anything that avoids
unnecessary compression/decompression at this point in time would also
be useful.

I know that for unix based os's that deleting a file while there are
still file handles open to it, doesn't create a problem. The file
handles remain valid, but the file no longer appears on the filesystem.
The blocks it uses don't get freed up until the last file handle
accessing it is closed.

Question is, how portable is that behaviour?

i.e. the following script works as described above on linux:

#!/usr/bin/python
import os

tmpfile = open("/tmp/testfile","w+")
tmpfile.write("This is just a test file\n")

fd = open("/tmp/testfile","r")

tmpfile.write("line 2 to show that you can still write to it\n")

os.remove("/tmp/testfile")

if not os.path.exists("/tmp/testfile"):
    tmpfile.write("line 3 writen even though the file has been
deleted\n")

tmpfile.close()

# now read it in
line = fd.read()
while line:
    print line
    line = fd.read()

fd.close()


--
Regards,
Darragh Bailey

Systems Software Engineer
Hewlett Packard Galway Ltd.
+353 91 75-4674

> -----Original Message-----
> From: Brian Warner [mailto:warner-buildbot at lothar.com]
> Sent: 18 February 2007 00:41
> To: Bailey, Darragh
> Cc: Buildbot-devel at lists.sourceforge.net
> Subject: Re: [Buildbot-devel] build log files
> 
> "Bailey, Darragh" <dbailey at hp.com> writes:
> 
> > Noticed that my build logs take up a lot of space. Since I have 64
> > builds running and the largest log is at 11MB, this can rapidly grow
to
> > take a lot of space. On average, this looks to be about 300MB per
day.
> >
> > So I'm wondering if anyone has done any work on supporting
compressed
> > log files in buildbot. I would like to keep about 2 week's worth of
log
> > files. But not if it's going to take up 4GB of space.
> 
> Hm, that's a good idea. It isn't trivial to implement, since the
logfiles
> are
> stored in a slightly weird way (designed to allow incremental-append
while
> other people are reading from them, and to accomodate multiple
"channels"
> like stdout vs stderr), but it isn't an impossible task.
> 
> I've created (trac) ticket #26 for this issue.
> 
> > Would this be something worthwhile adding to buildbot in general or
do I
> > just have unusually large log files?
> 
> I think is *would* be worthwhile to add. I've found that my own
> buildmasters
> are consuming a lot of space with logfiles too. My usual approach is
to
> use a
> crontab to prune the logs after a few weeks (note that you can prune
the
> logs
> while leaving the rest of the build status in place). But I suspect
that
> my
> builds are shorter and run less frequently than yours.
> 
> cheers,
>  -Brian