[Buildbot-devel] slow buildbot startup

[Vitali Lovich]

From looking at the log line, I think the messages about the load pickle is not relevant.
If you look at the timestamps it’s quick.  I think there’s something that’s outside the builder that’s taking so long.

I’m on OS X so no strace but I’ll take a look & see if anything stands out in dtruss.

-Vitali

> On Jun 5, 2015, at 7:49 PM, Dustin J. Mitchell <dustin at v.igoro.us> wrote:
> 
> Can you get an strace or equivalent measurement of hwat it's doing for
> those several seconds per builder?
> 
> At a guess, it's opening a bunch of pickle files, but that's not
> normal behavior, so I'd like to confirm it.
> 
> Dustin
> 
> On Fri, Jun 5, 2015 at 2:29 AM, Vitali Lovich <vlovich at gmail.com> wrote:
>> Hi,
>> 
>> I’ve noticed that over time our buildbot takes a really long time to start.
>> At this point it now takes roughly a minute to get to “BuildMaster is
>> running”.
>> 
>> I’ve noticed what look to be substantive delays as soon as it hits loading
>> the builders:
>> 
>> 15839 2015-06-04 23:19:52-0700 [-] Setting up database with URL
>> 'sqlite:///state.sqlite'
>> 15840 2015-06-04 23:19:52-0700 [-] setting database journal mode to 'wal'
>> 15841 2015-06-04 23:19:52-0700 [-] adding 5 new slaves, removing 0
>> 15842 2015-06-04 23:19:52-0700 [-] adding 14 new builders, removing 0
>> 15843 2015-06-04 23:19:52-0700 [-] trying to load status pickle from
>> /var/buildbot/apple-ci/master/workdir/A/builder
>> 15844 2015-06-04 23:19:52-0700 [-] added builder A in category
>> 15845 2015-06-04 23:20:03-0700 [-] trying to load status pickle from
>> /var/buildbot/apple-ci/master/workdir/B/builder
>> 15846 2015-06-04 23:20:03-0700 [-] added builder B in category
>> 15847 2015-06-04 23:20:03-0700 [-] trying to load status pickle from
>> /var/buildbot/apple-ci/master/workdir/C/builder
>> 15848 2015-06-04 23:20:03-0700 [-] added builder C in category
>> 15849 2015-06-04 23:20:12-0700 [-] trying to load status pickle from
>> /var/buildbot/apple-ci/master/workdir/D/builder
>> 15850 2015-06-04 23:20:12-0700 [-] added builder D in category
>> 15851 2015-06-04 23:20:17-0700 [-] trying to load status pickle from
>> /var/buildbot/apple-ci/master/workdir/E/builder
>> 15852 2015-06-04 23:20:17-0700 [-] added builder E in category
>> 15853 2015-06-04 23:20:17-0700 [-] trying to load status pickle from
>> /var/buildbot/apple-ci/master/workdir/F/builder
>> 15854 2015-06-04 23:20:17-0700 [-] added builder F Stage in category
>> 
>> We can see adding 6 builders here takes ~25 seconds.
>> 
>> Any suggestions (or tips on how to profile it) are welcome.
>> 
>> Thanks,
>> Vitali
>> 
>> ------------------------------------------------------------------------------
>> 
>> _______________________________________________
>> Buildbot-devel mailing list
>> Buildbot-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>