[Buildbot-devel] Inconsistent UI performance?

[David Coppit]

Sorry for the shot-in-the-dark question...

We've set up a UI responsiveness monitor on the same machine as our
master. We load the UI once every minute. Less than 1% of the time the
response takes more than 10 seconds, sometimes as long as 60 or 120
seconds.

We're concerned that this will become a bigger issue as load, builders,
builds, history, etc. increase in the future. We're also concerned that
we've somehow compromised the performance of our master.

Can anyone confirm whether this is "normal behavior"?

We've dumped stacks for threads during one of these stalls and saw that
the waterfall page is pretty slow. It appears to be reading a lot of pickle
files. But we haven't done page rendering measurements to confirm, and the
threading model makes us wonder if we're even catching a good snapshot of
the state at the time of the stall.

Here are some stats:

Version: Heavily modified 0.8.1
Hardware: Enterprise-class Dell server. 12 GB RAM. Load < .25
Slaves: about 30.

buildrequests: 29k rows
builds: 31k rows
buildset_properties: 17k rows
buildsets: 8k rows
changes: 90k rows
scheduler_changes: 160k rows (I believe there is a fix after 0.8.1)
schedulers: 19 (2 nightly, 1 try, 16 a subclass of AnyBranch)
sourcestamp_changes: 38k rows
sourcestamps: 8k rows
submissions: 18k rows

Two logs part of every build are around 5mb in size.

Build outputs are being stored on a separate filer that is mounted on the
slaves and master.