[Buildbot-devel] Inconsistent UI performance?

[David Coppit]

I forgot to mention that there are 45k pickle files (Files of the form
"<builder name>/<build number>".)

On Tue, 8 Feb 2011, David Coppit wrote:

> Sorry for the shot-in-the-dark question...
>
> We've set up a UI responsiveness monitor on the same machine as our
> master. We load the UI once every minute. Less than 1% of the time the
> response takes more than 10 seconds, sometimes as long as 60 or 120
> seconds.
>
> We're concerned that this will become a bigger issue as load, builders,
> builds, history, etc. increase in the future. We're also concerned that
> we've somehow compromised the performance of our master.
>
> Can anyone confirm whether this is "normal behavior"?
>
> We've dumped stacks for threads during one of these stalls and saw that
> the waterfall page is pretty slow. It appears to be reading a lot of pickle
> files. But we haven't done page rendering measurements to confirm, and the
> threading model makes us wonder if we're even catching a good snapshot of
> the state at the time of the stall.
>
> Here are some stats:
>
> Version: Heavily modified 0.8.1
> Hardware: Enterprise-class Dell server. 12 GB RAM. Load < .25
> Slaves: about 30.
>
> buildrequests: 29k rows
> builds: 31k rows
> buildset_properties: 17k rows
> buildsets: 8k rows
> changes: 90k rows
> scheduler_changes: 160k rows (I believe there is a fix after 0.8.1)
> schedulers: 19 (2 nightly, 1 try, 16 a subclass of AnyBranch)
> sourcestamp_changes: 38k rows
> sourcestamps: 8k rows
> submissions: 18k rows
>
> Two logs part of every build are around 5mb in size.
>
> Build outputs are being stored on a separate filer that is mounted on the
> slaves and master.
>

_____________________________________________________________________
David Coppit                                       http://coppit.org/