low amount of apache connection but slow server

[daruosha]

I know these topics had been discussed thousand of times on this forum and I gone through so many of them, but i'm still stocked with a slow server and now obvious reason:

one of our cPanel servers has very inconstant ping time and slow response. Below is the output of command i've tried to find the the issue.

Ping result from a another server in the same datacenter (in normal cases it has to be 1ms)

Pinging xxxxxx.com [xxx.xxx.xxx.xxx] with 32 bytes of data:
Reply from 5.: bytes=32 time=94ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=4ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=1ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=212ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=51ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=72ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=809ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=102ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=27ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=136ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=2ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=104ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=37ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=1ms TTL=63
Reply from xxx.xxx.xxx.xxx: bytes=32 time=26ms TTL=63

Ping statistics for xxx.xxx.xxx.xxx:
    Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 809ms, Average = 111ms

-bash-3.2# netstat -nt | grep :80 | wc -l
127

top c 
top - 21:24:33 up 1 day,  1:37,  3 users,  load average: 43.40, 23.00, 16.57
Tasks: 348 total,  14 running, 332 sleeping,   0 stopped,   2 zombie
Cpu(s): 13.0%us, 12.4%sy,  0.0%ni, 53.8%id, 19.9%wa,  0.1%hi,  0.9%si,  0.0%st
Mem:  12298904k total, 11224360k used,  1074544k free,   540368k buffers
Swap: 16771852k total,      144k used, 16771708k free,  6909828k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                             
21833 hhariri   16   0  231m  53m 6968 S 16.4  0.4   0:00.55 /usr/bin/php /home/hhariri/public_html/index.php                     
21850 mansoore  18   0  188m  30m  11m R 10.8  0.3   0:00.33 /usr/bin/php /home/mansoore/public_html/wp-login.php                 
21852 lnggts    18   0  214m  36m 6836 R 10.8  0.3   0:00.33 /usr/bin/php /home/lnggts/public_html/index.php                      
21845 goldis    16   0  185m  42m  31m R 10.5  0.4   0:00.32 /usr/bin/php /home/goldis/public_html/index.php                      
21853 lnggts    18   0  210m  33m 6820 R  9.8  0.3   0:00.30 /usr/bin/php /home/lnggts/public_html/index.php   
.....

-bash-3.2# iostat
Linux 2.6.18-348.12.1.el5 (webhosting01.aivivid.com) 	10/15/2013

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.14    0.54   16.29   15.59    0.00   56.44

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              78.48      2416.16      3348.41  223085388  309159733
sda1              0.00         0.02         0.00       1969          4
sda2              1.46         2.70       130.33     249123   12033497
sda3              0.00         0.02         0.00       1480        288
sda4              0.00         0.00         0.00          8          0
sda5             77.01      2413.42      3218.07  222832346  297125944

-bash-3.2# cat /etc/my.cnf 
[mysqld]
innodb_file_per_table=1
open_files_limit=33740


max_connections 	= 300
wait_timeout 		= 28800
interactive_timeout 	= 28800
query_cache_size	= 32M
join_buffer_size 	= 512K
thread_cache_size	= 4
table_cache 		= 500
innodb_buffer_pool_size = 512M
key_buffer_size 	= 3G
tmp_table_size 		= 128M
max_heap_table_size 	= 128M
-bash-3.2#

Apache Parameters:
fileetag: All
keepalive: On
keepalivetimeout: 2
maxclients: 150
maxkeepaliverequests: 100
maxrequestsperchild: 2000
maxspareservers: 25
minspareservers: 20
root_options: ExecCGI, FollowSymLinks, IncludesNOEXEC, SymLinksIfOwnerMatch
serverlimit: 256
serversignature: Off
servertokens: Full
sslciphersuite: ALL:!ADH:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
startservers: 20
timeout: 50
traceenable: Off

Server Spec:
Dual Xeon Quad Core 2.13 / 12GB RAM / Running on ESXi Hypervisor 5.0


I guess it's due to high I/O latency (it has 15 snapshots in the ESXi; but in the meantime of writing this post, i'm consolidating snapshots).

Looking for your suggestion.

 

[cPanelMichael]

Hello

The following thread may be helpful to you:

Troubleshooting High Loads On Linux Servers

Thank you.

 

[thinkbot]

based on ping

check dmesg and /var/log/messages
if nothing is there related kernel/network card issues

then you have some hardware (network card) problems or network problems in datacenter for this particular server

 

[daruosha]

Ok mates, thanks all for your help.

As it turned out, we had a I/O mess on that server and after consolidating all snapshots of the VM in the ESXi host, everything seems
normal. I tweaked Apache configuration a bit to speed up the responses.