[Buildbot-devel] `make clean' is too slow on buildbot

[Weongyo Jeong]

On Fri, Apr 23, 2010 at 02:18:40PM -0400, Mark Roddy wrote:
> On Fri, Apr 23, 2010 at 2:09 PM, Weongyo Jeong <weongyo.jeong at gmail.com> wrote:
> > On Fri, Apr 23, 2010 at 10:30:38AM -0700, Dustin J. Mitchell wrote:
> >> On Thu, Apr 22, 2010 at 4:33 PM, Weongyo Jeong <weongyo.jeong at gmail.com> wrote:
> >> > It takes 32 seconds if I read it correctly. ??In real it normally took 2
> >> > ~ 3 seconds that server is enough fast having 4 core CPU and 8 GB mem.
> >> > Is there any reason?
> >>
> >> Buildbot doesn't do anything unusual that would cause slowness.
> >>
> >> Any number of things could cause this -- virtual machine clock skew?
> >
> > What virtual machine means? ?VMware or VirtualBox? ?Buildbot didn't run
> > on virtual machine. ?It ran on Dell server.
> >
> >> Maybe the 'make clean' in that instance actually had a lot of files to
> >> remove?
> >
> > One thing I noticed during `make clean' is that `make' process consumes
> > 100% CPU at a moment. ?But as you know make(1) operations are related
> > with mostly disk I/O so it doesn't make sense to me that it consumes a
> > lot of CPU.
> >
> >> Maybe those files are on an NFS mount that was busy at the
> >> time?
> >
> > No it's not on NFS mount point. ?Normally the disks are very idle.
> >
> >> Memory or CPU contention?
> >
> > No. ?I think it has enough memory and CPU cores.
> >
> 
> Is there anything else running on the server?  'make clean' is likely
> to be largely IO bound so if the server is performing other tasks as
> the same time (especially IO bound ones) it could just be the case
> that you're thrashing and/or fighting for the disk.

Maybe no.  The following are a server information which runs buildbot.
The server is almost idle.

[xxx at eng ~]$ uptime
 11:46:50 up 3 days, 21:55, 11 users,  load average: 0.00, 0.19, 0.28

[xxx at eng ~]$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 189392 514344 199080 6373244    0    0    11    28   12   18  1  0 99  0  0
 0  0 189392 514368 199080 6373244    0    0     0   108 1038 4684  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0     0 1004 4596  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0    12 1009 4655  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0     8 1008 4640  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0     0 1003 4648  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0     0 1004 4639  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0     0 1005 4644  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0    36 1019 4659  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0     8 1008 4583  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0     0 1004 4643  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0     0 1003 4643  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0     0 1004 4635  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0    20 1014 4661  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0     8 1009 4640  0  0 100  0  0
 0  0 189392 514368 199080 6373244    0    0     0     0 1005 4650  0  0 100  0  0

top - 11:51:09 up 3 days, 21:59, 11 users,  load average: 0.08, 0.10, 0.21
Tasks: 159 total,   1 running, 157 sleeping,   1 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8178372k total,  7664996k used,   513376k free,   199084k buffers
Swap: 16771840k total,   189392k used, 16582448k free,  6373312k cached

regards,
Weongyo Jeong