[Buildbot-devel] FW: ­Buildbot buildmaster hard­ware recommendations

[Fabrice Coulon]

Hi,

We have implemented our own configuration web UI for Buildbot the same web
UI is used instead of accessing directly the Buildbot process Webstatus
pages, but we use a XML-RPC and JSON API to ask Buildbot for build
summaries and links to the build and test reports, I believe that this is
where the bottleneck is located, we should get this data from the Buildbot
database and the pickle files on disk instead of asking the Buildbot
process to do that for us.

Thanks for the very useful info!


On Thu, Jun 13, 2013 at 1:48 PM, <vasslitvinov at pisem.net> wrote:

> Hi Fabrice,
>
> We're running our buildmaster on somewhat powerful H/W in terms of
> computatinal power:
>
> 24 GB RAM, 4.5 GHz Xeon (16 cores total), hug HDD (not SSD... not sure how
> to get physical params of the HDD, though), gigabit ethernet.
>
> Our load seems to be similar to yours - we have about ~90 buildslaves
> under control which run several hundreds of builders.
>
>
> So far the most improvement we saw was when we switched from 0.8.0 to
> 0.8.7p1 (I believe it were this cache improvements you refer + some
> multithreading support added in 0.8.3).
>
> BTW our workaround (plus easing users life) was to implement custom web UI
> that runs in separate process (thus no GIL conflict with Buildbot) and goes
> almost directly to underlying database and pickle files - we use
> Buildbot-provided API to connect to DB but then we do raw SQL queries.
>
> Also we tend to avoid heavy pages like unfiltered waterfall or (even more
> heavy) unfiltered grid.
>
>
> Hope that helps.
>
>
> Чтв 13 Июн 2013 13:10:42 +0400, Fabrice Coulon <coruja074 at gmail.com>
> написал:
>
> I have been reading about this performance ticket:
> http://trac.buildbot.net/ticket/2246,
> a LRU cache fix that could boost the performance of the Buildbot Webstatus
> pages.
>
> It looks like this is not included in 0.8.6p1, so I may have to patch it
> myself, is this the
> "optimizations in the LRU cache performance" that you have been writing
> about in one
> previous email?
>
>
>
> --
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20130617/a543e564/attachment.html>