[Buildbot-devel] Maximum log size

[Chris AtLee]

Hi everyone,

We've been running into a few problems where runaway processes
generate HUGE log files (several hundred MB, sometimes a few GB),
which causes the master to become unresponsive and then eventually
crash or be killed by the oom-killer in the kernel.

I'm looking at one way of addressing this, which is to be able to set
an upper limit on the size of the log.  I've got a basic
implementation working [1], but ran into a few things where I wasn't
sure what the right thing to do was.

- Should the HEADER channel be included in the maximum size?
Currently I'm including it in the total size of the log, but not
preventing additional entries made to the HEADER channel from being
logged.  I've considered adding another counter that keeps track of
everything in the log except headers, so that the header channel
doesn't count towards the maximum log size.

- I'm not truncating individual chunks.  If the log file is under the
maximum size, the entire chunk will get logged, otherwise the chunk
won't get logged at all, with the exception of HEADER chunks.

- If the log is truncated, should the step be terminated at that time?
 I would guess no, but I'd to know what the community thinks.

- Should some portion of tail of the log be kept even if the log is truncated?

Cheers,
Chris

[1] http://github.com/catlee/buildbot/commits/maxlogsize