[Buildbot-devel] Several scalability questions

[David Coppit]

On Jan 18, 2011, at 2:27 PM, tom fogal wrote:

> David Coppit <dcoppit at nvidia.com> writes:
> [snip]
>> It alludes to some limit on the number of slaves attached to a single
>> master. Is 40 slaves considered to be "a lot"? What are the symptoms
>> of having too many slaves?
> 
> I don't know personally, but I think the mozilla && chrome teams are
> using much larger systems.  e.g. on:
> 
>  http://trac.buildbot.net/wiki/SuccessStories
> 
> mozilla says they have "700 build and test slaves".
> 

I'm guessing they're running multiple masters. :) I'm looking for some guidance on when to go to multiple masters.

>> It appears from watching the system execute that the UI can
>> slow down sch eduling and vice-versa. We thought that BB was
>> multi-threaded. Can it handle hundreds of pageviews along with
>> scheduling, builders, and custom build steps on the master? Is it
>> possible that we've serialized the threads through table locks or
>> something? (See question #6 below)
> 
> Would threading even really help?  Python's GIL is the source of
> much... contention. :P
> 
> I think the situation has improved somewhat in python 3, but don't
> quote me on that.

Good pointer! Not being Python gurus we didn't think to blame the interpreter.  This presentation by David Beazley (of SWIG fame) says that multithreading actually produces *worse* performance:

http://www.dabeaz.com/python/UnderstandingGIL.pdf

He says that Python 3.2 has a new GIL implementation that helps some, but not at all for I/O situations. Our machine has something like 24 cores, so we may be massively shooting ourselves in our feet. We should try dropping down to 1 thread.

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------