[Buildbot-devel] LogLineObserver and really long lines

[Greg Ward]

Hi --

I'm playing around with LogObserver and LogLineObserver, and it looks
to me as though LogLineObserver does not handle very long log lines
well.  Specifically, I'm running a ShellCommand that simply does "cat
/tmp/make.out" where /tmp/make.out is a real build log containing one
very long "javac" command (over 20k).  I have implemented two
LogObservers:
  * CountingObserver, which subclasses LogObserver directly and simply
    counts the number of bytes and number of newline chars seen in all calls
    to outReceived()
  * ErrorObserver, which is supposed to scan for "poison pill" error
messages, but
    currently also counts bytes and lines in outLineReceived()

CountingObserver counts 28906 bytes and 72 newline chars, which is
exactly what /tmp/make.out contains.  But ErrorObserver only counts
5757 bytes and 36 lines.  It looks like that huge 20k command line is
being passed to CountingObserver.outReceived() in several 4k chunks,
but ErrorObserver simply never sees most of the lines in those chunks.
 It only gets the last line of the last chunk.  Or something like
that, I think.

>From the code, it's clear that LogLineObserver is heavily dependent on
twisted.protocol.basic.LineOnlyReceiver ... so is LineOnlyReceiver
generally known to be broken in this way?  I'm using Twisted 2.5.0
under Python 2.5.1 and a moderately patched Buildbot 0.7.6.

Greg