[Buildbot] #3272: Buildbot start-up time starts to tip-over once there logs for ~3000 builds in a builder (especially if a build has lots of steps)

[Buildbot trac]

#3272: Buildbot start-up time starts to tip-over once there logs for ~3000 builds
in a builder (especially if a build has lots of steps)
------------------------+--------------------
Reporter:  vlovich      |       Owner:
    Type:  defect       |      Status:  new
Priority:  major        |   Milestone:  0.8.x
 Version:  0.8.10       |  Resolution:
Keywords:  performance  |
------------------------+--------------------

Comment (by tardyp):

 I had the same problem in my prod. I made a quick hack to fix the issue,
 which I did not upstreamed. I prefer working on long term solution with
 buildbot nine.

 There are several scaling limitations on eight.

 - It stores a lot of files in the same diretory (all files related to a
 builder's dir is stored in one dir, which is problematic on several
 filesystem)
 a good solution for this is to store them in multiple directories (encoded
 with the lower builds of the build number)
 - determineNextBuildNumber is fundamentally flawed as it parsed all the
 build files at startup. A good solution for this would be to store it
 either in the builder pickle itself, or in its own file (so that one can
 override the next buildnumber easily)

--
Ticket URL: <http://trac.buildbot.net/ticket/3272#comment:6>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation