[Buildbot-devel] Several scalability questions

[Dustin J. Mitchell]

On Tue, Jan 18, 2011 at 1:48 PM, David Coppit <dcoppit at nvidia.com> wrote:
>> mozilla says they have "700 build and test slaves".
>
> I'm guessing they're running multiple masters. :) I'm looking for some guidance on when to go to multiple masters.

Note the situation has evolved.  At my last count, we had 1145 slaves.

>> I think the situation has improved somewhat in python 3, but don't
>> quote me on that.
>
> Good pointer! Not being Python gurus we didn't think to blame the interpreter.  This presentation by David Beazley (of SWIG fame) says that multithreading actually produces *worse* performance:
>
> http://www.dabeaz.com/python/UnderstandingGIL.pdf

That's why Tom described it as "contentious" - Dave happens to think
that the GIL should perform better and that Twisted Python in a bad
thing.  I (and the Twisted folks) think otherwise.  You just have to
exercise some caution when writing twisted applications - and Buildbot
hasn't been too cautious :(

> He says that Python 3.2 has a new GIL implementation that helps some, but not at all for I/O situations. Our machine has something like 24 cores, so we may be massively shooting ourselves in our feet. We should try dropping down to 1 thread.

Well, especially when you're running a Twisted app like Buildbot,
which only uses threads for blocking operations (e.g., database
access), you will never saturate more than one core, so you can send
those other 23 back :)

Dustin