[Buildbot-devel] Compressing build logs, input wanted

[Axel Hecht]

2009/1/20 Marcus Lindblom <macke at yar.nu>

> Axel Hecht wrote:
> > Hi all,
> >
> > I just pushed a git branch,
> > http://github.com/Pike/buildbot/commits/bug26, to compress logs
> > post-facto with bz2.
> >
> > Right now, it does so rather unconditionally, with a hidden pref in
> > LogFile.compressLogs. I guess there would be some performance hits on
> > the master for large logs, see my comments in
> > http://buildbot.net/trac/ticket/26#comment:16.
> >
> > I'd like to know what others would expect, and who's relying on what in
> > the logfiles.
>
> We're just reading them as text, so no comment on that part.
>
> It seems to make sense to compress post-facto in a separate thread, then
> do an atomic rename (from log.bz2.part to log.bz2)


I could probably just return the deferred from finish() and use that in the
tests.


> However, wouldn't it make more sense to gzip them, so that they could be
> sent in compressed format directly to browsers that accepted gzip format?


Not so much as most users look at the HTML formatted output of the logs, and
those need to get the html wraps fed upfront, and of course you wouldn't be
able to just compress the logfile, but you had to compress the logs inside
the channels, which is going to be harder.

And you can't just update the master dir by compressing the logfiles
manually, which some folks might want to do.


> I'd also wish for pluggable compression (so that I could use 7z, which
> is fast and I like it :)


The downside of pluggable compression would be that you'd have to try all
compression modes on reading logs, and on hasContent etc.


Re Jean-Paul, http://twistedmatrix.com/trac/ticket/727 is about twistd.log,
this is about the build log stdout etc files.

Axel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20090120/1d0d35f7/attachment.html>