[Buildbot] #3654: Really stark performance regression at startup and later on
Buildbot trac
trac at buildbot.net
Sun Dec 11 17:25:01 UTC 2016
#3654: Really stark performance regression at startup and later on
---------------------+-----------------------
Reporter: kayhayen | Owner:
Type: defect | Status: new
Priority: major | Milestone: undecided
Version: 0.9.1 | Keywords:
---------------------+-----------------------
This is the log of a startup.
{{{
2016-12-11 18:10:58+0100 [-] Loading buildbot.tac...
2016-12-11 18:11:08+0100 [-] Loaded.
2016-12-11 18:11:08+0100 [-] twistd 16.6.0 (/home/nuitka-
buildmaster/sandbox/bin
/python 2.7.9) starting up.
2016-12-11 18:11:08+0100 [-] reactor class:
twisted.internet.epollreactor.EPollR
eactor.
2016-12-11 18:11:08+0100 [-] Starting BuildMaster -- buildbot.version:
0.9.1
2016-12-11 18:11:09+0100 [-] Loading configuration from '/home/nuitka-
buildmaste
r/master/master.cfg'
2016-12-11 18:11:11+0100 [-] Setting up database with URL
'sqlite:/state.sqlite'
2016-12-11 18:11:11+0100 [-] setting database journal mode to 'wal'
2016-12-11 18:11:11+0100 [-] doing housekeeping for master 1
bigbrother:/home/nu
itka-buildmaster/master
2016-12-11 18:11:17+0100 [-] doing housekeeping for master 1
bigbrother:/home/nu
itka-buildmaster/master
2016-12-11 18:11:17+0100 [-] adding 2 new changesources, removing 0
2016-12-11 18:11:17+0100 [-] gitpoller: using workdir '/home/nuitka-
buildmaster/master/gitpoller-public'
2016-12-11 18:11:18+0100 [-] PBServerFactory starting on 9999
2016-12-11 18:11:18+0100 [-] Starting factory
<twisted.spread.pb.PBServerFactory instance at 0xb42439b8>
2016-12-11 18:11:18+0100 [-] adding 132 new builders, removing 0
2016-12-11 18:12:38+0100 [-] adding 18 new schedulers, removing 0
2016-12-11 18:12:48+0100 [-] Periodic scheduler <biweekly>: missed
scheduled build time - building immediately
2016-12-11 18:12:51+0100 [-] initializing www plugin 'waterfall_view'
2016-12-11 18:12:51+0100 [-] initializing www plugin 'console_view'
2016-12-11 18:12:51+0100 [-] RotateLogSite starting on 8010
2016-12-11 18:12:51+0100 [-] Starting factory
<buildbot.www.service.RotateLogSite instance at 0xb3f9d968>
2016-12-11 18:12:51+0100 [-] Setting up http.log rotating 10 files of
10000000 bytes each
2016-12-11 18:12:51+0100 [-] adding 7 new workers, removing 0
2016-12-11 18:12:51+0100 [-] PBServerFactory starting on 9989
2016-12-11 18:12:51+0100 [-] Starting factory
<twisted.spread.pb.PBServerFactory instance at 0xb3fab058>
2016-12-11 18:12:57+0100 [-] BuildMaster is running
}}}
I am pretty sure swapping is not the issue. The housekeeping job takes
multiple seconds. I recall it did this when it was new database too. The
strangest part is how slow adding builders appears to be, more than 90
seconds.
I used to use 0.8.x without ever noticing performance issues on the same
machine, with more services running. This was only 2 minutes, I have seen
far worse startups leading me to override systemd startup time for the
buildbot master.
When doing queries in the web interface, e.g. listing builders, it too
feels *very* slow. It easily takes more than a second for the Builders to
load which nothing appears in twisted.log that would explain what the
master is doing.
The state DB is currently 31MB if that means anything.
Yours,
Kay
--
Ticket URL: <http://trac.buildbot.net/ticket/3654>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation
More information about the bugs
mailing list