[users at bb.net] buildbot CPU usage
Neil Gilmore
ngilmore at grammatech.com
Fri Aug 26 15:05:57 UTC 2016
I'm currently looking at a step with 3 logs:
81531 lines
489285 lines
489311 lines
An earlier successful run would have that first log at 244080 lines.
This particular build is stuck, though. :( (which is why I'm looking at
it.) (A bit off-topic, but I tried stopping it. The last step is marked
cancelled, but that's the only effect.)
Neil Gilmore
grammatech.com
On 8/26/2016 6:25 AM, Pierre Tardy wrote:
>
> The ram looks like sufficient, it might be a good test to try and
> increase the number of cpu for that VM.
> In your trace, I can see the use of up to 7 threads at the same time,
> so you might gain by going 8 CPUs
>
> Also, make sure that your VM host is not overbooked. In my experience
> of using VMware VMs provided by IT, overbooking has been a source of
> inexplicable performance issues.
>
> 12k lines is a lot, but buildbot shall support this kind of load
> without issue.
>
> Le ven. 26 août 2016 à 12:05, Francesco Di Mizio
> <francescodimizio at gmail.com <mailto:francescodimizio at gmail.com>> a écrit :
>
> It's a vmware virtual machine with 4 GIGs RAM and 4 CPUs at 3Ghz. It
> runs, among other marginal things, 2 docker containers - one for
> the buildbot and one for the postgres db.
>
> The most beefy logs have around 12K lines. Is it too much?
> Also some other logs are read from the worker's filesystem and
> added as additional logs.
>
> On Fri, Aug 26, 2016 at 11:51 AM, Pierre Tardy <tardyp at gmail.com
> <mailto:tardyp at gmail.com>> wrote:
>
> Cool!
> I can indeed see 3 spikes.
>
> Looks related to logs and logs compression.
>
> What is the HW spec of your master machine?
> How much log does your build generate?
>
> Pierre
>
> Le ven. 26 août 2016 à 11:42, Francesco Di Mizio
> <francescodimizio at gmail.com
> <mailto:francescodimizio at gmail.com>> a écrit :
>
> Pierre,
>
> I enabled it, waited 1 min and saw the spike, then stopped
> after a few secs. Attached the json.
> Awesome tool btw, work wonders!
>
> On Thu, Aug 25, 2016 at 1:14 PM, Pierre Tardy
> <tardyp at gmail.com <mailto:tardyp at gmail.com>> wrote:
>
> You can try to hit the button in the morning with a
> two hours gather period, and hope that you see the
> spike during that period..
>
>
> Le jeu. 25 août 2016 à 12:17, Francesco Di Mizio
> <francescodimizio at gmail.com
> <mailto:francescodimizio at gmail.com>> a écrit :
>
> Thanks a lot! Pierre I will def will give it a
> shot. I am not sure I'll be able to smash that
> 'start recording' button as the UI isusuallystuck
> when the CPU spikes. Updates to come!
>
> On Thu, Aug 25, 2016 at 10:45 AM, Pierre Tardy
> <tardyp at gmail.com <mailto:tardyp at gmail.com>> wrote:
>
> Hi Francesco,
>
> I spent some time in order to implement a
> profiler plugin for buildbot
>
> You can give it a look, and send your
> profile.json file if you need more analysis
> from me.
> https://github.com/tardyp/buildbot_profiler
>
> Regards,
> Pierre
>
>
> Le mer. 24 août 2016 à 22:43, Francesco Di
> Mizio <francescodimizio at gmail.com
> <mailto:francescodimizio at gmail.com>> a écrit :
>
> I've tried and it's not an easy task
> because of my Win into Vagrant into Docker
> setup.
> I'll try again soon when I get a Linux box!
>
> On Fri, Aug 19, 2016 at 5:54 PM, Vasily
> <just.one.man at yandex.ru
> <mailto:just.one.man at yandex.ru>> wrote:
>
> Umm, no. VTune has Python support
> starting 2017 Beta, and, well, it was
> my team (at Intel) work actually :-)
>
> P.S. I'm from Intel, too.
>
> Thanks,
> Vasily
>
> 19 авг. 2016 г. 18:17 пользователь
> "Francesco Di Mizio"
> <francescodimizio at gmail.com
> <mailto:francescodimizio at gmail.com>>
> написал:
>
> I had thought you were making fun
> of Intel somehow ;)
>
>
> On Aug 19, 2016 5:07 PM, "Pierre
> Tardy" <tardyp at gmail.com
> <mailto:tardyp at gmail.com>> wrote:
>
> ahah
>
> I though this was a taunt on
> me being employed by Intel.
> I actually had mitigated
> experience with vtune few
> years ago, and didn't know
> they had python support until
> then.
> Being an opensource guy, I
> usually neglegate to look at
> proprietary stuff.
>
> Pierre
>
> Le ven. 19 août 2016 à 12:18,
> Vasily <just.one.man at yandex.ru
> <mailto:just.one.man at yandex.ru>>
> a écrit :
>
> I'm again suggesting to
> look into Python profiling
> capabilities of Intel®
> VTune™ Amplifier. It could
> run statistical profiling
> for a long time and
> display CPU usage over
> time, so the developer can
> look at specific time
> range where CPU usage was
> too high and see which
> functions were executed.
>
> Thanks,
> Vasily
>
> 19 авг. 2016 г. 11:57
> пользователь "Pierre
> Tardy" <tardyp at gmail.com
> <mailto:tardyp at gmail.com>>
> написал:
>
> Hi Francesco,
>
> Your described setup
> looks sane to me.
>
> The problems we are
> trying to catch are
> cpu spikes, as far as
> I understand, which
> does not happen for
> very long, but are
> very annoying for
> users, as it is
> blocking the reactor.
>
> This problem is not
> easy to see in the
> profile you sent, as
> this profile is over
> long time, so we see
> the average of each
> method during the day
> and not the spikes.
>
> What would really be
> needed is a on-demand
> profiler which would
> detect cpu spikes and
> only log the stack
> traces during those times.
>
> Here is a nice blog
> pst explaining why
> statistic profiling is
> cool and easy to
> implement in python.
> https://nylas.com/blog/performance
>
> For 0.9.1 I want to
> concentrate on
> scalability, and write
> a debugging ui plugin
> based on those ideas
> (and probably code)
>
> That would be great if
> your team can help on
> that matter.
>
> Regards,
> Pierre
>
>
>
>
>
>
>
>
> _______________________________________________
> users mailing list
> users at buildbot.net
> https://lists.buildbot.net/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildbot.net/pipermail/users/attachments/20160826/d0465aca/attachment.html>
More information about the users
mailing list