[Buildbot-devel] Limitations using buildbot for building a huge amount of projects

[Iago Toral Quiroga]

Hi,

we are using buildbot to provide continuous integration of the gnome
platform (http://buildbot.gnome.org). Basically, we run several Build
Master instances and provide an html frontend that renders the main page
getting information from each master.

We have some problems with this solution though:

1.- We need a port per master instance that must be accessible to
external slave machines. We are building a large list of modules (~200)
so this means we need to open ~200 ports in the firewall.

2.- We would like the solution to be scalable, being able to add more
slaves. Unfortunately, adding more slaves implies to multiply the number
of required connections by the number of slaves, which does not seem to
be a scalable solution.

A mate of mine, John Carr, has tried to address the first problem by
using a Socks proxy, so only the socks server proxy port would be
public. Unfortunately this makes the second problem even worse, for we
need twice as many connections (slave<->proxy and proxy<->master). 

He also suggested to implement a custom ITransport so that masters and
slaves would not open new TCP connections but use a shared one instead,
which in principle would be good solution to both problems.

I would like to know what buildbot developers think about this... What
do you think about this ITransport solution? do you have any good/better
solution? do you have any plans for future buildbot releases that would
fix this issue or help with it? etc

Thanks in advance!

Iago