[Buildbot-devel] WebStatus caching (Was: Re: Few buildbot improvements)

[Marcus Lindblom]

On 2010-04-10 23:39, Marcus Lindblom wrote:
> On 2010-04-10 22:24, Dustin J. Mitchell wrote:
>
>> I do want to be careful that these additional UI features do not
>> drastically increase the load on the buildmaster.  All web rendering
>> is essentially done synchronously, so lots of HTTP requests can starve
>> the buildmaster's attention to slaves.  Chris's work on slave-side log
>> buffering reduces the impact of this problem somewhat, but not
>> entirely.

[snip]

> My BuildbotIcon does a lot of polling too, and our developer-intranet
> page has 5 buildmasters which are polled every 10-15 secs (to show on
> the public screen). This polling adds up time.
>
> Adding caching of HTML inside buildbot would've been cool, but I'm not
> too familiar with buildbot's internals and good cache invalidation
> strategies (which is Hard (tm) see quote[3]).

I just got an idea for a relatively simple way to do some caching and 
speed-up the WebStatus.

Simply cache each generated page (i.e. a dict from URLs to (html, time) 
tuples) for 5-15 secs (configurable). That should:

  * cut down CPU load quite a lot during many requests to the top pages 
(i.e. waterfall)
  * not use too much memory (the last 10 sec of pages can't be _that_ 
much, if we exclude logs)
  * still provide a reasonable user experience (5-15s update freq on 
non-log pages is pretty ok, I think)
  * be short enough so that manual cache invalidation is not necessary
  * allow for cache invalidation and longer timeouts in the future (if a 
build changes status, strip all pages related to that build from the 
cache, reduce expiration time for "global" pages to 5 secs or less)

As I'm new to caching schemes, I might've missed something.

Comments?

Cheers,
/Marcus