[Buildbot-devel] Maximum log size

[Axel Hecht]

Hey Chris,

I'm curious, what can go so wrong that we're running into this problem? This
might be interesting as defense in depth, but the fix is probably elsewhere.

For your patch:

I think the config variable should be called maxLogSize instead of
logMaxSize.

I don't think that you should worry about header chunks, it's more about
stdout vs stderr.

If we drop chunks and let the command run on, we should drop chunks in the
middle of the log, IMHO.

I think I favour having at least the option to kill the command if the log
grows beyond the limit, in our case, that's probably what we want anyway.
Dropping output sounds like an error condition either way, and should be
signaled as a build failure. You could add an option abortOnLargeLog =
WARN/ERROR/EXCEPTION.

Axel

2009/9/15 Chris AtLee <chris at atlee.ca>

> Hi everyone,
>
> We've been running into a few problems where runaway processes
> generate HUGE log files (several hundred MB, sometimes a few GB),
> which causes the master to become unresponsive and then eventually
> crash or be killed by the oom-killer in the kernel.
>
> I'm looking at one way of addressing this, which is to be able to set
> an upper limit on the size of the log.  I've got a basic
> implementation working [1], but ran into a few things where I wasn't
> sure what the right thing to do was.
>
> - Should the HEADER channel be included in the maximum size?
> Currently I'm including it in the total size of the log, but not
> preventing additional entries made to the HEADER channel from being
> logged.  I've considered adding another counter that keeps track of
> everything in the log except headers, so that the header channel
> doesn't count towards the maximum log size.
>
> - I'm not truncating individual chunks.  If the log file is under the
> maximum size, the entire chunk will get logged, otherwise the chunk
> won't get logged at all, with the exception of HEADER chunks.
>
> - If the log is truncated, should the step be terminated at that time?
>  I would guess no, but I'd to know what the community thinks.
>
> - Should some portion of tail of the log be kept even if the log is
> truncated?
>
> Cheers,
> Chris
>
> [1] http://github.com/catlee/buildbot/commits/maxlogsize
>
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry® Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9-12, 2009. Register now!
> http://p.sf.net/sfu/devconf
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20090922/c711db4c/attachment.html>