[Buildbot-devel] review request: #459

[Dustin J. Mitchell]

On Mon, Mar 23, 2009 at 5:15 AM, Marcus Lindblom <macke at yar.nu> wrote:
> I couldn't find anything obviosly wrong, but I had to take most of it on
> faith. (I could understand that you used a big weakref dict, and an
> limited-size LRU with strong refs, but not much more.. ) I really need to
> learn more about the rest of the framework to draw any significant
> conclusions.

It's all structured over the BuilderStatus > BuildStatus >
BuildStepStatus > LogFile containment hierarchy.  There are a finite
number of builders, which must be resident in memory at all times.
There are potentially an infinite number of builds, however.  For each
build, there are a finite, small number of steps, and each step has a
small number of logfiles.

So this patch takes two approaches.  First, to limit memory
consumption, each builder only keeps references to a few in-memory
Build objects.  As you mention, this uses a weakref dict and an LRU
list, to avoid loading a build from disk when it is already resident
in memory.

Second, to limit disk storage, the Builder periodically "prunes"
itself.  This entails skimming through the builder's basedir, which
contains three kinds of files
  * builder (pickle of the builder object)
  * ### (pickle of a numbered build object)
  * ###-log-* (pickle of a Logfile)
The pruning process deletes Logfile pickles older than the logHorizon
and build pickles older than the buildHorizon.  In doing so, it skips
any builds that are in memory, as they are clearly in use.

This process avoids unpickling "old" objects, as doing so would be
prohibitively slow.  However, this leads to a situation where a build
may reference logfiles that are no longer available on-disk.  The
BuildStepStatus's checkLogfiles checks this situation when a build is
loaded, and removes any references to deleted logfiles at that time.

I hope this helps.  I will go ahead and commit.

Dustin

-- 
Storage Software Engineer
http://www.zmanda.com