[Buildbot] #3272: Buildbot start-up time starts to tip-over once there logs for ~3000 builds in a builder (especially if a build has lots of steps)

[Buildbot trac]

#3272: Buildbot start-up time starts to tip-over once there logs for ~3000 builds
in a builder (especially if a build has lots of steps)
------------------------+------------------------
Reporter:  vlovich      |       Owner:
    Type:  defect       |      Status:  new
Priority:  major        |   Milestone:  undecided
 Version:  0.8.10       |  Resolution:
Keywords:  performance  |
------------------------+------------------------

Comment (by vlovich):

 First, if the build number were to be a folder instead that contains the
 other files, that would cut this down to an O(M) problem.

 If the last next builder number were to be stored in the DB, it would an
 O(1) in the common case.  On startup, if the build exists, search up until
 you find the next one.  If it doesn't, search down until you find the one
 that does.

 Even if the number were to be recorded in the DB though there would still
 need to be a cleanup to provide some more structure.  To that end, I would
 recommend that build logs are re-organized to live under:
 YYYYMM/DD/Build #/...

 That will keep the number of files in a given directory to be much more
 manageable.  Probably would need to keep a information in the DB to get at
 the YYYYMM/DD information given a build #.

 Alternatively, it might be possible to accomplish improvements with a more
 simple change.  Simply have some kind of nesting value (probably
 configurable but default to something big like 5000 or 10000).  That
 controls how many builds at a given level.

 Even if only 1 level is implemented, at 10 000 you're looking at 100
 million builds with logfiles still around before you have to look at
 tackling supporting more levels (or re-architecting again).

--
Ticket URL: <http://trac.buildbot.net/ticket/3272#comment:1>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation