[Buildbot] #3654: Really stark performance regression at startup and later on

Buildbot trac trac at buildbot.net
Sun Dec 11 17:25:01 UTC 2016


#3654: Really stark performance regression at startup and later on
---------------------+-----------------------
Reporter:  kayhayen  |      Owner:
    Type:  defect    |     Status:  new
Priority:  major     |  Milestone:  undecided
 Version:  0.9.1     |   Keywords:
---------------------+-----------------------
 This is the log of a startup.

 {{{

 2016-12-11 18:10:58+0100 [-] Loading buildbot.tac...
 2016-12-11 18:11:08+0100 [-] Loaded.
 2016-12-11 18:11:08+0100 [-] twistd 16.6.0 (/home/nuitka-
 buildmaster/sandbox/bin
 /python 2.7.9) starting up.
 2016-12-11 18:11:08+0100 [-] reactor class:
 twisted.internet.epollreactor.EPollR
 eactor.
 2016-12-11 18:11:08+0100 [-] Starting BuildMaster -- buildbot.version:
 0.9.1
 2016-12-11 18:11:09+0100 [-] Loading configuration from '/home/nuitka-
 buildmaste
 r/master/master.cfg'
 2016-12-11 18:11:11+0100 [-] Setting up database with URL
 'sqlite:/state.sqlite'
 2016-12-11 18:11:11+0100 [-] setting database journal mode to 'wal'
 2016-12-11 18:11:11+0100 [-] doing housekeeping for master 1
 bigbrother:/home/nu
 itka-buildmaster/master
 2016-12-11 18:11:17+0100 [-] doing housekeeping for master 1
 bigbrother:/home/nu
 itka-buildmaster/master
 2016-12-11 18:11:17+0100 [-] adding 2 new changesources, removing 0
 2016-12-11 18:11:17+0100 [-] gitpoller: using workdir '/home/nuitka-
 buildmaster/master/gitpoller-public'
 2016-12-11 18:11:18+0100 [-] PBServerFactory starting on 9999
 2016-12-11 18:11:18+0100 [-] Starting factory
 <twisted.spread.pb.PBServerFactory instance at 0xb42439b8>
 2016-12-11 18:11:18+0100 [-] adding 132 new builders, removing 0
 2016-12-11 18:12:38+0100 [-] adding 18 new schedulers, removing 0
 2016-12-11 18:12:48+0100 [-] Periodic scheduler <biweekly>: missed
 scheduled build time - building immediately
 2016-12-11 18:12:51+0100 [-] initializing www plugin 'waterfall_view'
 2016-12-11 18:12:51+0100 [-] initializing www plugin 'console_view'
 2016-12-11 18:12:51+0100 [-] RotateLogSite starting on 8010
 2016-12-11 18:12:51+0100 [-] Starting factory
 <buildbot.www.service.RotateLogSite instance at 0xb3f9d968>
 2016-12-11 18:12:51+0100 [-] Setting up http.log rotating 10 files of
 10000000 bytes each
 2016-12-11 18:12:51+0100 [-] adding 7 new workers, removing 0
 2016-12-11 18:12:51+0100 [-] PBServerFactory starting on 9989
 2016-12-11 18:12:51+0100 [-] Starting factory
 <twisted.spread.pb.PBServerFactory instance at 0xb3fab058>
 2016-12-11 18:12:57+0100 [-] BuildMaster is running

 }}}

 I am pretty sure swapping is not the issue. The housekeeping job takes
 multiple seconds. I recall it did this when it was new database too. The
 strangest part is how slow adding builders appears to be, more than 90
 seconds.

 I used to use 0.8.x without ever noticing performance issues on the same
 machine, with more services running. This was only 2 minutes, I have seen
 far worse startups leading me to override systemd startup time for the
 buildbot master.

 When doing queries in the web interface, e.g. listing builders, it too
 feels *very* slow. It easily takes more than a second for the Builders to
 load which nothing appears in twisted.log that would explain what the
 master is doing.

 The state DB is currently 31MB if that means anything.

 Yours,
 Kay

--
Ticket URL: <http://trac.buildbot.net/ticket/3654>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation


More information about the bugs mailing list