[Buildbot] #3607: /raw log download endpoint uses a huge amount of memory

[Buildbot trac]

#3607: /raw log download endpoint uses a huge amount of memory
---------------------+-------------------
Reporter:  aomighty  |      Owner:
    Type:  defect    |     Status:  new
Priority:  major     |  Milestone:  0.9.+
 Version:  master    |   Keywords:
---------------------+-------------------
 Hi,

 I'm using buildbot on a project that generates fairly large build logs
 (65MB of logs, around 300,000 lines). I have noticed that when I use the
 /raw endpoint (http://MASTERURL/api/v2/logs/BUILDID/raw), the memory usage
 spikes while the master sends the logs. Although the logs are 65MB, the
 memory used to process the request goes much higher than this, and I see
 the virtual memory of the buildbot process increasing by about 700MB,
 causing the server to OOM. As a workaround, I gave the server a large swap
 file, but the memory usage still seems excessive given the log file size.

 Looking at the code, it appears the relevant lines are in
 '''master/buildbot/db/logs.py'''. It appears that each line is stored as a
 compressed row in the database, retrieved in order, and slurped into
 memory. Using 10x more memory than the actual logfile seems rather high
 even for this approach, but I may be missing something.

--
Ticket URL: <http://trac.buildbot.net/ticket/3607>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation