[Buildbot-devel] slow buildbot startup

[Vitali Lovich]

crapton of getdirentries64 calls.  So it looks like it’s stat’ing a lot of things.

There’s ~ 108556 files in a builder so my guess is that’s the cause.

Not sure why it’s trying to stat through all the log files though.

-Vitali

> On Jun 6, 2015, at 1:07 AM, Vitali Lovich <vlovich at gmail.com> wrote:
> 
> From looking at the log line, I think the messages about the load pickle is not relevant.
> If you look at the timestamps it’s quick.  I think there’s something that’s outside the builder that’s taking so long.
> 
> I’m on OS X so no strace but I’ll take a look & see if anything stands out in dtruss.
> 
> -Vitali
> 
>> On Jun 5, 2015, at 7:49 PM, Dustin J. Mitchell <dustin at v.igoro.us> wrote:
>> 
>> Can you get an strace or equivalent measurement of hwat it's doing for
>> those several seconds per builder?
>> 
>> At a guess, it's opening a bunch of pickle files, but that's not
>> normal behavior, so I'd like to confirm it.
>> 
>> Dustin
>> 
>> On Fri, Jun 5, 2015 at 2:29 AM, Vitali Lovich <vlovich at gmail.com> wrote:
>>> Hi,
>>> 
>>> I’ve noticed that over time our buildbot takes a really long time to start.
>>> At this point it now takes roughly a minute to get to “BuildMaster is
>>> running”.
>>> 
>>> I’ve noticed what look to be substantive delays as soon as it hits loading
>>> the builders:
>>> 
>>> 15839 2015-06-04 23:19:52-0700 [-] Setting up database with URL
>>> 'sqlite:///state.sqlite'
>>> 15840 2015-06-04 23:19:52-0700 [-] setting database journal mode to 'wal'
>>> 15841 2015-06-04 23:19:52-0700 [-] adding 5 new slaves, removing 0
>>> 15842 2015-06-04 23:19:52-0700 [-] adding 14 new builders, removing 0
>>> 15843 2015-06-04 23:19:52-0700 [-] trying to load status pickle from
>>> /var/buildbot/apple-ci/master/workdir/A/builder
>>> 15844 2015-06-04 23:19:52-0700 [-] added builder A in category
>>> 15845 2015-06-04 23:20:03-0700 [-] trying to load status pickle from
>>> /var/buildbot/apple-ci/master/workdir/B/builder
>>> 15846 2015-06-04 23:20:03-0700 [-] added builder B in category
>>> 15847 2015-06-04 23:20:03-0700 [-] trying to load status pickle from
>>> /var/buildbot/apple-ci/master/workdir/C/builder
>>> 15848 2015-06-04 23:20:03-0700 [-] added builder C in category
>>> 15849 2015-06-04 23:20:12-0700 [-] trying to load status pickle from
>>> /var/buildbot/apple-ci/master/workdir/D/builder
>>> 15850 2015-06-04 23:20:12-0700 [-] added builder D in category
>>> 15851 2015-06-04 23:20:17-0700 [-] trying to load status pickle from
>>> /var/buildbot/apple-ci/master/workdir/E/builder
>>> 15852 2015-06-04 23:20:17-0700 [-] added builder E in category
>>> 15853 2015-06-04 23:20:17-0700 [-] trying to load status pickle from
>>> /var/buildbot/apple-ci/master/workdir/F/builder
>>> 15854 2015-06-04 23:20:17-0700 [-] added builder F Stage in category
>>> 
>>> We can see adding 6 builders here takes ~25 seconds.
>>> 
>>> Any suggestions (or tips on how to profile it) are welcome.
>>> 
>>> Thanks,
>>> Vitali
>>> 
>>> ------------------------------------------------------------------------------
>>> 
>>> _______________________________________________
>>> Buildbot-devel mailing list
>>> Buildbot-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>> 
>