[Buildbot-devel] Limitations using buildbot for building a huge amount of projects

[Iago Toral Quiroga]

Hi Juan-Paul!

I added John to the CC. He is been working on this too and will be
interested.

First, thanks for your comments, I really appreciate your help. I answer
your questions below...

El jue, 24-04-2008 a las 09:51 -0400, Jean-Paul Calderone escribió:
> On Thu, 24 Apr 2008 13:33:41 +0200, Iago Toral Quiroga <itoral at igalia.com> wrote:
> >Hi,
> >
> >we are using buildbot to provide continuous integration of the gnome
> >platform (http://buildbot.gnome.org). Basically, we run several Build
> >Master instances and provide an html frontend that renders the main page
> >getting information from each master.
> 
> Can you elaborate on why you have more than one master?
> 

We need to build around 200 projects, and we want to do each of them in
several machines too, having a waterfall view of the build results for
each of the slave machines we have use to build the project. For
example:

http://build.gnome.org/libxml2/

That's project libxml2 building on a Debian/Sid slave machine (first
builder) and a Red Hat slave machine (second builder).

To achieve this we have one master per project, and one builder per
slave machine. Then, in each slave machine we have a Build Slave
instance per project. This way we have a nice waterfall view of the
build results of each project (each master provides its own, only for
the project it is building). So, we just added a main page that queries
the state of each master instance, providing a summary for all the
projects.

If there is a way to achieve this using a single master instance I'd be
more than glad to know about that, it would make my life way easier :)

> >
> >We have some problems with this solution though:
> >
> >1.- We need a port per master instance that must be accessible to
> >external slave machines. We are building a large list of modules (~200)
> >so this means we need to open ~200 ports in the firewall.
> 
> It'd probably be possible to tunnel all these connections over a single
> port going through the firewall and then demultiplex them on the other
> side.  This would take a bit of coding, at least, but nothing too
> revolutionary.  Only necessary if you stick with multiple masters, of
> course.

We tried this already, using Socks to (de)multiplex connections and it
works fine AFAIK. I also tried to do it from scratch and managed to get
it done, but ran into problems with twisted complaining with
BananaError("Security precaution: List too long.")... The problem
though, is the number of connections we need as I explain below.

> >2.- We would like the solution to be scalable, being able to add more
> >slaves. Unfortunately, adding more slaves implies to multiply the number
> >of required connections by the number of slaves, which does not seem to
> >be a scalable solution.
> >
> >A mate of mine, John Carr, has tried to address the first problem by
> >using a Socks proxy, so only the socks server proxy port would be
> >public. Unfortunately this makes the second problem even worse, for we
> >need twice as many connections (slave<->proxy and proxy<->master).
> 
> 
> Twisted easily scales to many thousands of connections.  Are you worried
> about the ability of the masters to handle all the connections?  The
> ability of the firewall to?  Something else?

We are getting this error message in twistd.log:

2008/04/24 16:20 +0200 [twisted.spread.pb.PBServerFactory] Could not
accept new connection (EMFILE)

> >He also suggested to implement a custom ITransport so that masters and
> >slaves would not open new TCP connections but use a shared one instead,
> >which in principle would be good solution to both problems.
> 
> One approach here would be to use SSH as the transport and use a channel
> for each logical connection.

That could be a way to go... depending on the amount of channels that
SSH transport allows. Otherwise we would be running into the same
problem again I guess.

> >I would like to know what buildbot developers think about this... What
> >do you think about this ITransport solution? do you have any good/better
> >solution? do you have any plans for future buildbot releases that would
> >fix this issue or help with it? etc
> 
> I'd still like to hear more about the problem. :)
> Jean-Paul
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
> Don't miss this year's exciting event. There's still time to save $100. 
> Use priority code J8TL2D2. 
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>