[Buildbot-devel] Buildbot buildmaster hardware recommendations

[Pierre Tardy]

On Wed, Jun 12, 2013 at 10:34 AM, Fabrice Coulon <coruja074 at gmail.com>wrote:

> Hi,
>
> I am using Buildbot at my work place since 2010. The system has
> problem to scale up i.e, the Buildbot web status is very slow (it can
> take
> sometimes one minute to retrieve results from it under heavy load).
>

Hi Fabrice,
Nice to see a new team using buildbot with a big setup. Is this
inapropriate to ask for your company name/domain of activity?

Buildbot 0.8 has indeed a scalability problem. Buildbot nine is our roadmap
to really fix this, feel free to participate to the effort.

Meanwhile, there might be a few tricks to try and make it work better.

- Your log/pickle directory needs to be fast, ssd is indeed a good idea, my
server has raid0 10krpm disks, which looks okay.
- make sure you dont have too much files in the pickle dir. ext4 tends to
explode if there are too much files in the same dir. you can:
   - Switch to reiserfs, which is supposed to be better for lot of file in
same dir
   - remove old builds sooner (there is an option too let buildbot do this
automatically)
   - I personnally did hack buildbot to spread the logs in several
directories per builder, but my setup is a bit special, as I have one
builder which has lots lots of builds.
- run statprof in prod:
  https://pypi.python.org/pypi/statprof/
I did run that under manhole (   c['manhole'] =
manhole.AuthorizedKeysManhole(1234, "authorized_keys") ), and did find out
a few issues in my custom steps that uses a lot of CPU.
- also make sure you are using latest version of buildbot, there has been
some optimizations in the LRU cache performance since 2010. (I had to
backport it)

Regards,
Pierre
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20130612/39b664d7/attachment.html>