[Buildbot-devel] Limitations using buildbot for building a huge amount of projects

Iago Toral Quiroga itoral at igalia.com
Thu Apr 24 14:57:51 UTC 2008

Hi Juan-Paul!

I added John to the CC. He is been working on this too and will be

First, thanks for your comments, I really appreciate your help. I answer
your questions below...

El jue, 24-04-2008 a las 09:51 -0400, Jean-Paul Calderone escribió:
> On Thu, 24 Apr 2008 13:33:41 +0200, Iago Toral Quiroga <itoral at igalia.com> wrote:
> >Hi,
> >
> >we are using buildbot to provide continuous integration of the gnome
> >platform (http://buildbot.gnome.org). Basically, we run several Build
> >Master instances and provide an html frontend that renders the main page
> >getting information from each master.
> Can you elaborate on why you have more than one master?

We need to build around 200 projects, and we want to do each of them in
several machines too, having a waterfall view of the build results for
each of the slave machines we have use to build the project. For


That's project libxml2 building on a Debian/Sid slave machine (first
builder) and a Red Hat slave machine (second builder).

To achieve this we have one master per project, and one builder per
slave machine. Then, in each slave machine we have a Build Slave
instance per project. This way we have a nice waterfall view of the
build results of each project (each master provides its own, only for
the project it is building). So, we just added a main page that queries
the state of each master instance, providing a summary for all the

If there is a way to achieve this using a single master instance I'd be
more than glad to know about that, it would make my life way easier :)

> >
> >We have some problems with this solution though:
> >
> >1.- We need a port per master instance that must be accessible to
> >external slave machines. We are building a large list of modules (~200)
> >so this means we need to open ~200 ports in the firewall.
> It'd probably be possible to tunnel all these connections over a single
> port going through the firewall and then demultiplex them on the other
> side.  This would take a bit of coding, at least, but nothing too
> revolutionary.  Only necessary if you stick with multiple masters, of
> course.

We tried this already, using Socks to (de)multiplex connections and it
works fine AFAIK. I also tried to do it from scratch and managed to get
it done, but ran into problems with twisted complaining with
BananaError("Security precaution: List too long.")... The problem
though, is the number of connections we need as I explain below.

> >2.- We would like the solution to be scalable, being able to add more
> >slaves. Unfortunately, adding more slaves implies to multiply the number
> >of required connections by the number of slaves, which does not seem to
> >be a scalable solution.
> >
> >A mate of mine, John Carr, has tried to address the first problem by
> >using a Socks proxy, so only the socks server proxy port would be
> >public. Unfortunately this makes the second problem even worse, for we
> >need twice as many connections (slave<->proxy and proxy<->master).
> Twisted easily scales to many thousands of connections.  Are you worried
> about the ability of the masters to handle all the connections?  The
> ability of the firewall to?  Something else?

We are getting this error message in twistd.log:

2008/04/24 16:20 +0200 [twisted.spread.pb.PBServerFactory] Could not
accept new connection (EMFILE)

> >He also suggested to implement a custom ITransport so that masters and
> >slaves would not open new TCP connections but use a shared one instead,
> >which in principle would be good solution to both problems.
> One approach here would be to use SSH as the transport and use a channel
> for each logical connection.

That could be a way to go... depending on the amount of channels that
SSH transport allows. Otherwise we would be running into the same
problem again I guess.

> >I would like to know what buildbot developers think about this... What
> >do you think about this ITransport solution? do you have any good/better
> >solution? do you have any plans for future buildbot releases that would
> >fix this issue or help with it? etc
> I'd still like to hear more about the problem. :)
> Jean-Paul
> -------------------------------------------------------------------------
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
> Don't miss this year's exciting event. There's still time to save $100. 
> Use priority code J8TL2D2. 
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel

More information about the devel mailing list