[Buildbot-devel] review request: #459

[Marcus Lindblom]

Dustin J. Mitchell wrote:
> On Mon, Mar 23, 2009 at 5:15 AM, Marcus Lindblom <macke at yar.nu> wrote:
>   
>> I couldn't find anything obviosly wrong, but I had to take most of it on
>> faith. (I could understand that you used a big weakref dict, and an
>> limited-size LRU with strong refs, but not much more.. ) I really need to
>> learn more about the rest of the framework to draw any significant
>> conclusions.
>>     
>
> It's all structured over the BuilderStatus > BuildStatus >
> BuildStepStatus > LogFile containment hierarchy.  There are a finite
> number of builders, which must be resident in memory at all times.
> There are potentially an infinite number of builds, however.  For each
> build, there are a finite, small number of steps, and each step has a
> small number of logfiles.
>
> So this patch takes two approaches.  First, to limit memory
> consumption, each builder only keeps references to a few in-memory
> Build objects.  As you mention, this uses a weakref dict and an LRU
> list, to avoid loading a build from disk when it is already resident
> in memory.
>
> Second, to limit disk storage, the Builder periodically "prunes"
> itself.  This entails skimming through the builder's basedir, which
> contains three kinds of files
>   * builder (pickle of the builder object)
>   * ### (pickle of a numbered build object)
>   * ###-log-* (pickle of a Logfile)
> The pruning process deletes Logfile pickles older than the logHorizon
> and build pickles older than the buildHorizon.  In doing so, it skips
> any builds that are in memory, as they are clearly in use.
>
> This process avoids unpickling "old" objects, as doing so would be
> prohibitively slow.  However, this leads to a situation where a build
> may reference logfiles that are no longer available on-disk.  The
> BuildStepStatus's checkLogfiles checks this situation when a build is
> loaded, and removes any references to deleted logfiles at that time.
>
> I hope this helps.  I will go ahead and commit.
>   
Ok. Thanks. That makes sense (it cleared up a bit more in my head after 
I wrote the above too).

Old logfiles could always be snapshotted/backed up if one would like 
that to keep them for ages, so I think it's all good... :)

Cheers,
/Marcus