[Buildbot-devel] Inconsistent UI performance?

[David Coppit]

On Tue, 8 Feb 2011, Dustin J. Mitchell wrote:

> On Tue, Feb 8, 2011 at 4:21 PM, Amber Yust <ayust at yelp.com> wrote:
>> Buildbot is single threaded (but asynchronous). This means that when the
>> master has lots of load due to a new build request or the like, it will
>> temporarily slow down UI responsiveness due to other tasks being processed
>> and eating up slices of processing time.
>> There are plans to move the UI to a separate process, so that Buildbot build
>> processing would not have an impact on UI load times, but that has not yet
>> been implemented.
>
> Amber's right - this is a well-known problem with buildbot.  The
> waterfall and other "aggregate" views are particularly awful, because
> they block *any* other activity on the master until they are done
> rendering.
>
> Phillipe's suggestion is a good one - you can set the various build
> horizons to something small enough, and the build cache to something
> large enough, that your master doesn't thrash loading pickles all day.
>
> In the long term, the two fixes to this are:
> 1. store build data in a database, not pickles
> 2. render complex pages client-side, using REST queries to get the data
>
> Both are slated for Buildbot-0.9.0.

Okay, so we can expect degraded performance as the history of builds
increases. I think the aggregate page we developed mostly hits the DB, so
we could disable the waterfall view and other pickle-file-heavy views.

We're also looking at optimizing waterfall to hit the pickle files less.
We implemented one fix to sorting. We are also looking at using DB data to
do pre-filtering for the time range before the pickle files have to be
read--I would think that would have the same effect as deleting the older
pickle files.

Were there similar optimizations already implemented since 0.8.1? Perhaps
we should work on upgrading to the latest version instead...

0.9.0 sounds great!

Thanks,
David

_____________________________________________________________________
David Coppit                                       http://coppit.org/