[Buildbot-devel] FW: Buildbot buildmaster hardware recommendations

[Fabrice Coulon]

Thank you for your quick answer Pierre!

The company is http://www.axis.com/, the business is "Network Cameras". We
have just made the big step between Buildbot 0.7.12 and Buildbot 0.8.6p1.

It looks like the memory consumption has exploded between 0.7.x and 0.8.x,
I was wondering also if there was a formula available to estimate the
memory consumed by one Buildmaster? Currently we do not know if the system
is leaking memory or just caching a lot of data (hoping that it will stop
growing when it will reach the maximum number of cached objects). I have
noticed that by default Buildbot caches many objects like for example 15
build requests per builders, but how big is for example build request
object, etc.






On Wed, Jun 12, 2013 at 2:12 PM, Fabrice Coulon
<fabrice.coulon at hotmail.com>wrote:

>
>
> ------------------------------
> Date: Wed, 12 Jun 2013 12:16:37 +0200
> Subject: Re: [Buildbot-devel] Buildbot buildmaster hardware recommendations
> From: tardyp at gmail.com
> To: coruja074 at gmail.com
> CC: buildbot-devel at lists.sourceforge.net
>
> On Wed, Jun 12, 2013 at 10:34 AM, Fabrice Coulon <coruja074 at gmail.com>wrote:
>
> Hi,
>
> I am using Buildbot at my work place since 2010. The system has
> problem to scale up i.e, the Buildbot web status is very slow (it can
> take
> sometimes one minute to retrieve results from it under heavy load).
>
>
> Hi Fabrice,
> Nice to see a new team using buildbot with a big setup. Is this
> inapropriate to ask for your company name/domain of activity?
>
> Buildbot 0.8 has indeed a scalability problem. Buildbot nine is our
> roadmap to really fix this, feel free to participate to the effort.
>
> Meanwhile, there might be a few tricks to try and make it work better.
>
> - Your log/pickle directory needs to be fast, ssd is indeed a good idea,
> my server has raid0 10krpm disks, which looks okay.
> - make sure you dont have too much files in the pickle dir. ext4 tends to
> explode if there are too much files in the same dir. you can:
>    - Switch to reiserfs, which is supposed to be better for lot of file in
> same dir
>    - remove old builds sooner (there is an option too let buildbot do this
> automatically)
>    - I personnally did hack buildbot to spread the logs in several
> directories per builder, but my setup is a bit special, as I have one
> builder which has lots lots of builds.
> - run statprof in prod:
>   https://pypi.python.org/pypi/statprof/
> I did run that under manhole (   c['manhole'] =
> manhole.AuthorizedKeysManhole(1234, "authorized_keys") ), and did find out
> a few issues in my custom steps that uses a lot of CPU.
> - also make sure you are using latest version of buildbot, there has been
> some optimizations in the LRU cache performance since 2010. (I had to
> backport it)
>
> Regards,
> Pierre
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20130612/cf2660cc/attachment.html>