[Buildbot-devel] Buildbot performance with 25+ slaves

[Benoît Allard]

Hi John

> -----Original Message-----
> 
> I've had around 20 builders running on roughly that many slaves for about a
> week and performance has been fine. These are intended to be our 'sanity'
> builds and run off a SingleBranchScheduler. Yesterday I added another ~20
> builders running on the same slaves but with different configuration to create
> a nightly 'release' process. Everything was fine. Today I also added a Try
> server with three platforms/builders. Problems began to occur with each of the
> builders showing that builds were pending but these were never being kicked
> off. The python process on the buildmaster was also sitting at 99.9%. I
> actually stuffed up when I added the Try server and attached it to the release
> builders. This was around the time we first started noticing the pending
> problem.

I have had intermittent problem of this kind.

See here: http://markmail.org/thread/dloqmqeeb44minji

The cause are still not known, but there is a possibility for an infinite loop to happen, blocking all the scheduling functionalities. This seems to happen on busy days, however, I've not been able to reproduce it for some months.

In my case, restarting the master, or canceling the blocking build (have to find it !) does solve it.

Note that this issue I had is still latent as nothing has been solved in this area.

That's why I would look at the option 1. In my case it helped that we are using postgresql and that it has it's own logs ... I don't know if sqlite can be configured this way, but I don't think the issue I saw is limited to postgresql.

Option 2 and 3 are out of question. Buildbot is meant to handle big load.

I've no idea about option 4.

Good luck !

Benoît.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6467 bytes
Desc: not available
URL: <http://buildbot.net/pipermail/devel/attachments/20121116/ca315993/attachment.bin>