[Buildbot-devel] Limitations using buildbot for building a huge amount of projects

[Jean-Paul Calderone]

On Thu, 24 Apr 2008 13:33:41 +0200, Iago Toral Quiroga <itoral at igalia.com> wrote:
>Hi,
>
>we are using buildbot to provide continuous integration of the gnome
>platform (http://buildbot.gnome.org). Basically, we run several Build
>Master instances and provide an html frontend that renders the main page
>getting information from each master.

Can you elaborate on why you have more than one master?

>
>We have some problems with this solution though:
>
>1.- We need a port per master instance that must be accessible to
>external slave machines. We are building a large list of modules (~200)
>so this means we need to open ~200 ports in the firewall.

It'd probably be possible to tunnel all these connections over a single
port going through the firewall and then demultiplex them on the other
side.  This would take a bit of coding, at least, but nothing too
revolutionary.  Only necessary if you stick with multiple masters, of
course.

>2.- We would like the solution to be scalable, being able to add more
>slaves. Unfortunately, adding more slaves implies to multiply the number
>of required connections by the number of slaves, which does not seem to
>be a scalable solution.
>
>A mate of mine, John Carr, has tried to address the first problem by
>using a Socks proxy, so only the socks server proxy port would be
>public. Unfortunately this makes the second problem even worse, for we
>need twice as many connections (slave<->proxy and proxy<->master).


Twisted easily scales to many thousands of connections.  Are you worried
about the ability of the masters to handle all the connections?  The
ability of the firewall to?  Something else?

>He also suggested to implement a custom ITransport so that masters and
>slaves would not open new TCP connections but use a shared one instead,
>which in principle would be good solution to both problems.

One approach here would be to use SSH as the transport and use a channel
for each logical connection.

>I would like to know what buildbot developers think about this... What
>do you think about this ITransport solution? do you have any good/better
>solution? do you have any plans for future buildbot releases that would
>fix this issue or help with it? etc

I'd still like to hear more about the problem. :)

Jean-Paul