[Buildbot-devel] gc.collect in BuilderStatus.prune

[Dustin J. Mitchell]

On Fri, Jun 17, 2011 at 12:56 PM, Imran Hameed <ihameed at imvu.com> wrote:
> We profiled buildbot (with cProfile and with a hastily-written
> sampling profiler) and found that:
>  - gc.collect, called from BuilderStatus.prune, consumes a significant
> amount of CPU time, and
>  - during periods of 100% CPU usage, we were typically evaluating
> BuilderStatus.prune.

Nicely done!

> Why is gc.collect called from BuilderStatus.prune? Is it necessary? I
> looked through status/builder.py's commit history and read
> http://trac.buildbot.net/ticket/459 (and
> http://trac.buildbot.net/ticket/458), but I am not yet familiar enough
> with buildbot's internals to know if
> aea14d3f245badd92c5e197803cf56cc4e6d96be's commit message or if the
> comments in bug #459 describe why gc.collect was added.

I don't have a good explanation, but perhaps catlee can help - that
was his commit.

> Are there other gotchas that we may encounter with our current setup?
> Has anyone else run into scalability issues with buildbot?

At Mozilla, we have, and catlee's been the one chasing them down.
We've seen a few things that may be related:
 - massive memory consumption that grows by big leaps many hours or
days after the master starts
 - long-lasting blocking operations (tens of seconds to minutes) that
lead to slave disconnections and general master slowness

The latter may be because of gc.collect calls; the former may be a
hint as to why calling gc.collect seemed like a good idea.

I'm copying Ben here, too, since we've talked about Buildbot's
performance problems recently.

Dustin