[users at bb.net] RAM consumption of twisted for buildbot 0.8.12?

[Pierre Tardy]

Hi Dan,
Buildbot eight manage a cache of unpickled builds, which is per builder
http://docs.buildbot.net/0.8.12/manual/cfg-global.html#caches
The doc here suggest the default cache is 500 builds, which looks very big
to me you can try to reduce that value
On my eight prod, I had to tweak that value a lot in order to compromise
speed VS memory.
The 3GB usage is indeed a common number (and often much more)

The pickle problem and its scalability issues is the main reason why we
started the nine project.

For the comparaison, I started a simple master.cfg with 2000 builders:
https://paste.fedoraproject.org/460567/89908147/

It runs happily on my mac with 150MB of memory.

Pierre

Le lun. 24 oct. 2016 à 16:53, Dan Kegel <dank at kegel.com> a écrit :

> 15 days of uptime, about 1450 builders, and twisted is using 1.7GB of
> *physical* ram.
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  1196 buildbot  20   0 3124m 1.7g 4456 S   10 14.8   1515:22 twistd
>
> Aw, what the heck, RAM is cheap these days... still, it's a bit surprising.
>
>
> On Mon, May 16, 2016 at 2:35 PM, Dan Kegel <dank at kegel.com> wrote:
> > What's normal RAM consumption for twisted on buildbot master for 0.8.12?
> > With 1359 builders, I'm seeing 1.4GB.  If memory use is proportional
> > to builders, that's 100MB/builder.
> >
> > Here's the output of top after two days of uptime.  This is steady
> > state; if someone clicks on the waterfall, CPU spikes to 277%, but ram
> > doesn't climb.
> >
> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >  1223 buildbot  20   0  2681m 1.4g  4340 D   0 17.7 130:18.76 twistd
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20161025/227c4052/attachment.html>