[Buildbot-devel] slow buildbot startup

[Vitali Lovich]

Opened 3272 with some analysis.  Basically the cost of the function is O(mn) since none of the logs are grouped by the build they’re in.
Thus you still have to iterate over all the logs.  My builder has ~3000 builds but ~100k files in the directory that it searches through.

-Vitali

> On Jun 6, 2015, at 1:32 AM, Vitali Lovich <vlovich at gmail.com> wrote:
> 
> Found the culprit I think: determineNextBuildNumber
> 
> -Vitali
> 
>> On Jun 6, 2015, at 1:12 AM, Vitali Lovich <vlovich at gmail.com> wrote:
>> 
>> crapton of getdirentries64 calls.  So it looks like it’s stat’ing a lot of things.
>> 
>> There’s ~ 108556 files in a builder so my guess is that’s the cause.
>> 
>> Not sure why it’s trying to stat through all the log files though.
>> 
>> -Vitali
>> 
>>> On Jun 6, 2015, at 1:07 AM, Vitali Lovich <vlovich at gmail.com> wrote:
>>> 
>>> From looking at the log line, I think the messages about the load pickle is not relevant.
>>> If you look at the timestamps it’s quick.  I think there’s something that’s outside the builder that’s taking so long.
>>> 
>>> I’m on OS X so no strace but I’ll take a look & see if anything stands out in dtruss.
>>> 
>>> -Vitali
>>> 
>>>> On Jun 5, 2015, at 7:49 PM, Dustin J. Mitchell <dustin at v.igoro.us> wrote:
>>>> 
>>>> Can you get an strace or equivalent measurement of hwat it's doing for
>>>> those several seconds per builder?
>>>> 
>>>> At a guess, it's opening a bunch of pickle files, but that's not
>>>> normal behavior, so I'd like to confirm it.
>>>> 
>>>> Dustin
>>>> 
>>>> On Fri, Jun 5, 2015 at 2:29 AM, Vitali Lovich <vlovich at gmail.com> wrote:
>>>>> Hi,
>>>>> 
>>>>> I’ve noticed that over time our buildbot takes a really long time to start.
>>>>> At this point it now takes roughly a minute to get to “BuildMaster is
>>>>> running”.
>>>>> 
>>>>> I’ve noticed what look to be substantive delays as soon as it hits loading
>>>>> the builders:
>>>>> 
>>>>> 15839 2015-06-04 23:19:52-0700 [-] Setting up database with URL
>>>>> 'sqlite:///state.sqlite'
>>>>> 15840 2015-06-04 23:19:52-0700 [-] setting database journal mode to 'wal'
>>>>> 15841 2015-06-04 23:19:52-0700 [-] adding 5 new slaves, removing 0
>>>>> 15842 2015-06-04 23:19:52-0700 [-] adding 14 new builders, removing 0
>>>>> 15843 2015-06-04 23:19:52-0700 [-] trying to load status pickle from
>>>>> /var/buildbot/apple-ci/master/workdir/A/builder
>>>>> 15844 2015-06-04 23:19:52-0700 [-] added builder A in category
>>>>> 15845 2015-06-04 23:20:03-0700 [-] trying to load status pickle from
>>>>> /var/buildbot/apple-ci/master/workdir/B/builder
>>>>> 15846 2015-06-04 23:20:03-0700 [-] added builder B in category
>>>>> 15847 2015-06-04 23:20:03-0700 [-] trying to load status pickle from
>>>>> /var/buildbot/apple-ci/master/workdir/C/builder
>>>>> 15848 2015-06-04 23:20:03-0700 [-] added builder C in category
>>>>> 15849 2015-06-04 23:20:12-0700 [-] trying to load status pickle from
>>>>> /var/buildbot/apple-ci/master/workdir/D/builder
>>>>> 15850 2015-06-04 23:20:12-0700 [-] added builder D in category
>>>>> 15851 2015-06-04 23:20:17-0700 [-] trying to load status pickle from
>>>>> /var/buildbot/apple-ci/master/workdir/E/builder
>>>>> 15852 2015-06-04 23:20:17-0700 [-] added builder E in category
>>>>> 15853 2015-06-04 23:20:17-0700 [-] trying to load status pickle from
>>>>> /var/buildbot/apple-ci/master/workdir/F/builder
>>>>> 15854 2015-06-04 23:20:17-0700 [-] added builder F Stage in category
>>>>> 
>>>>> We can see adding 6 builders here takes ~25 seconds.
>>>>> 
>>>>> Any suggestions (or tips on how to profile it) are welcome.
>>>>> 
>>>>> Thanks,
>>>>> Vitali
>>>>> 
>>>>> ------------------------------------------------------------------------------
>>>>> 
>>>>> _______________________________________________
>>>>> Buildbot-devel mailing list
>>>>> Buildbot-devel at lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/buildbot-devel
>>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://buildbot.net/pipermail/devel/attachments/20150606/eedcc103/attachment.html>