[Buildbot-devel] slow buildbot startup

[Dustin J. Mitchell]

Can you get an strace or equivalent measurement of hwat it's doing for
those several seconds per builder?

At a guess, it's opening a bunch of pickle files, but that's not
normal behavior, so I'd like to confirm it.

Dustin

On Fri, Jun 5, 2015 at 2:29 AM, Vitali Lovich <vlovich at gmail.com> wrote:
> Hi,
>
> I’ve noticed that over time our buildbot takes a really long time to start.
> At this point it now takes roughly a minute to get to “BuildMaster is
> running”.
>
> I’ve noticed what look to be substantive delays as soon as it hits loading
> the builders:
>
> 15839 2015-06-04 23:19:52-0700 [-] Setting up database with URL
> 'sqlite:///state.sqlite'
> 15840 2015-06-04 23:19:52-0700 [-] setting database journal mode to 'wal'
> 15841 2015-06-04 23:19:52-0700 [-] adding 5 new slaves, removing 0
> 15842 2015-06-04 23:19:52-0700 [-] adding 14 new builders, removing 0
> 15843 2015-06-04 23:19:52-0700 [-] trying to load status pickle from
> /var/buildbot/apple-ci/master/workdir/A/builder
> 15844 2015-06-04 23:19:52-0700 [-] added builder A in category
> 15845 2015-06-04 23:20:03-0700 [-] trying to load status pickle from
> /var/buildbot/apple-ci/master/workdir/B/builder
> 15846 2015-06-04 23:20:03-0700 [-] added builder B in category
> 15847 2015-06-04 23:20:03-0700 [-] trying to load status pickle from
> /var/buildbot/apple-ci/master/workdir/C/builder
> 15848 2015-06-04 23:20:03-0700 [-] added builder C in category
> 15849 2015-06-04 23:20:12-0700 [-] trying to load status pickle from
> /var/buildbot/apple-ci/master/workdir/D/builder
> 15850 2015-06-04 23:20:12-0700 [-] added builder D in category
> 15851 2015-06-04 23:20:17-0700 [-] trying to load status pickle from
> /var/buildbot/apple-ci/master/workdir/E/builder
> 15852 2015-06-04 23:20:17-0700 [-] added builder E in category
> 15853 2015-06-04 23:20:17-0700 [-] trying to load status pickle from
> /var/buildbot/apple-ci/master/workdir/F/builder
> 15854 2015-06-04 23:20:17-0700 [-] added builder F Stage in category
>
> We can see adding 6 builders here takes ~25 seconds.
>
> Any suggestions (or tips on how to profile it) are welcome.
>
> Thanks,
> Vitali
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Buildbot-devel mailing list
> Buildbot-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>