[Buildbot-devel] slow buildbot startup

[Vitali Lovich]

Found the culprit I think: determineNextBuildNumber

-Vitali

> On Jun 6, 2015, at 1:12 AM, Vitali Lovich <vlovich at gmail.com> wrote:
> 
> crapton of getdirentries64 calls.  So it looks like it’s stat’ing a lot of things.
> 
> There’s ~ 108556 files in a builder so my guess is that’s the cause.
> 
> Not sure why it’s trying to stat through all the log files though.
> 
> -Vitali
> 
>> On Jun 6, 2015, at 1:07 AM, Vitali Lovich <vlovich at gmail.com> wrote:
>> 
>> From looking at the log line, I think the messages about the load pickle is not relevant.
>> If you look at the timestamps it’s quick.  I think there’s something that’s outside the builder that’s taking so long.
>> 
>> I’m on OS X so no strace but I’ll take a look & see if anything stands out in dtruss.
>> 
>> -Vitali
>> 
>>> On Jun 5, 2015, at 7:49 PM, Dustin J. Mitchell <dustin at v.igoro.us> wrote:
>>> 
>>> Can you get an strace or equivalent measurement of hwat it's doing for
>>> those several seconds per builder?
>>> 
>>> At a guess, it's opening a bunch of pickle files, but that's not
>>> normal behavior, so I'd like to confirm it.
>>> 
>>> Dustin
>>> 
>>> On Fri, Jun 5, 2015 at 2:29 AM, Vitali Lovich <vlovich at gmail.com> wrote:
>>>> Hi,
>>>> 
>>>> I’ve noticed that over time our buildbot takes a really long time to start.
>>>> At this point it now takes roughly a minute to get to “BuildMaster is
>>>> running”.
>>>> 
>>>> I’ve noticed what look to be substantive delays as soon as it hits loading
>>>> the builders:
>>>> 
>>>> 15839 2015-06-04 23:19:52-0700 [-] Setting up database with URL
>>>> 'sqlite:///state.sqlite'
>>>> 15840 2015-06-04 23:19:52-0700 [-] setting database journal mode to 'wal'
>>>> 15841 2015-06-04 23:19:52-0700 [-] adding 5 new slaves, removing 0
>>>> 15842 2015-06-04 23:19:52-0700 [-] adding 14 new builders, removing 0
>>>> 15843 2015-06-04 23:19:52-0700 [-] trying to load status pickle from
>>>> /var/buildbot/apple-ci/master/workdir/A/builder
>>>> 15844 2015-06-04 23:19:52-0700 [-] added builder A in category
>>>> 15845 2015-06-04 23:20:03-0700 [-] trying to load status pickle from
>>>> /var/buildbot/apple-ci/master/workdir/B/builder
>>>> 15846 2015-06-04 23:20:03-0700 [-] added builder B in category
>>>> 15847 2015-06-04 23:20:03-0700 [-] trying to load status pickle from
>>>> /var/buildbot/apple-ci/master/workdir/C/builder
>>>> 15848 2015-06-04 23:20:03-0700 [-] added builder C in category
>>>> 15849 2015-06-04 23:20:12-0700 [-] trying to load status pickle from
>>>> /var/buildbot/apple-ci/master/workdir/D/builder
>>>> 15850 2015-06-04 23:20:12-0700 [-] added builder D in category
>>>> 15851 2015-06-04 23:20:17-0700 [-] trying to load status pickle from
>>>> /var/buildbot/apple-ci/master/workdir/E/builder
>>>> 15852 2015-06-04 23:20:17-0700 [-] added builder E in category
>>>> 15853 2015-06-04 23:20:17-0700 [-] trying to load status pickle from
>>>> /var/buildbot/apple-ci/master/workdir/F/builder
>>>> 15854 2015-06-04 23:20:17-0700 [-] added builder F Stage in category
>>>> 
>>>> We can see adding 6 builders here takes ~25 seconds.
>>>> 
>>>> Any suggestions (or tips on how to profile it) are welcome.
>>>> 
>>>> Thanks,
>>>> Vitali
>>>> 
>>>> ------------------------------------------------------------------------------
>>>> 
>>>> _______________________________________________
>>>> Buildbot-devel mailing list
>>>> Buildbot-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>> 
>> 
>