high load problem with high traffic on apache at nights

[LAZer]

hi , i recently faced a problem with my server load averages . sites traffic on my server increased recently more than 200000 page views daily is on some of the sites on my server . and i receive loads around 25.

every thing ( page load/download speed ) still are working normal with this high load because the server is a great one but the log softwares send me alerts about the high cpuload problem having a high load for more than 6-7 hours.

my server ram is 8 gb ddr II and it has 2* cuad core xeon cpus, the hards are raid10 and io rates are normal . the 2 things that are causing high load are apache connections and mysql connections which i optimized them the best i could . and that configuration was working fine for more than 4 5 monthes.

here is a sample of my top :

root@server2 [~]# w
 18:53:11 up 45 days,  9:00,  1 user,  load average: 18.19, 16.73, 15.50

at high load times there are lots of processes like this :

13182 nobody    15   0  125m  28m 3092 S  1.0  0.3   0:00.64 httpd
25322 nobody    15   0  125m  28m 3100 S  1.0  0.3   0:00.88 httpd
 1300 nobody    15   0  125m  28m 3092 S  0.7  0.3   0:01.33 httpd

nobody   22619  0.1  0.3 128612 28748 ?        S    18:59   0:00 /usr/local/apache/bin/httpd -k start -DSSL
nobody   22620  0.1  0.3 128612 28680 ?        S    18:59   0:00 /usr/local/apache/bin/httpd -k start -DSSL

 4203 xxx  16   0     0    0    0 Z 24.1  0.0   0:00.16 php <defunct>
 4205 xxx  16   0  131m  15m 9516 R 21.1  0.2   0:00.14 php
 4197 xxx  17   0  131m  15m 9516 R 15.1  0.2   0:00.10 php
 4204 xxx   16   0  130m  15m  10m S 12.1  0.2   0:00.08 php

xxx is the user name

mysql    11959 53.1  1.1 342080 98968 ?        Sl   18:28  20:33 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --log-error=/var/lib/mysql/server2..err --pid-file=/var/lib/mysql/server2.pid --socket=/var/lib/mysql/mysql.sock --port=3306

there is a little io waiting and memory swaping , also memory leaks are very low :

root@server2 [~]# iostat
Linux 2.6.18-194.32.1.el5xen (server2)       03/04/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          28.24    0.56   19.72    1.31    1.37   48.78

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
xvdb             39.60      2886.19        97.86 3834298211  130001032
xvdb1            39.60      2886.19        97.86 3834297763  130001032
xvda            162.98      1094.28      2236.21 1453748336 2970803498
xvda1             0.00         0.00         0.00       2724         98
xvda2           162.98      1094.28      2236.21 1453745412 2970803400
xvdc             25.56       850.02       319.08 1129245211  423897888
xvdc1            25.56       850.02       319.08 1129244763  423897888
dm-0            307.20      1094.28      2236.04 1453744330 2970576464
dm-1              0.02         0.00         0.17        736     226936
dm-2             54.91       850.02       319.08 1129244194  423897888
dm-3             52.31      2886.19        97.86 3834297322  130001032

root@server2 [~]# free -m
             total       used       free     shared    buffers     cached
Mem:          8192       7969        222          0        176       6109
-/+ buffers/cache:       1683       6508
Swap:         5087         82       5005

Apache Server Status for localhost

   Server Version: Apache/2.2.16 (Unix) mod_ssl/2.2.16
          OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4
          Perl/v5.8.8

   Server Built: Aug 15 2010 09:50:04
     _________________________________________________________________

   Current Time: Friday, 04-Mar-2011 19:12:50 IRST
   Restart Time: Friday, 04-Mar-2011 18:25:41 IRST
   Parent Server Generation: 0
   Server uptime: 47 minutes 9 seconds
   Total accesses: 346214 - Total Traffic: 307.0 GB
   CPU Usage: u384.42 s157.16 cu2403.91 cs0 - 134% CPU load
   122 requests/sec - 111.1 MB/second - 0.9 MB/request
   624 requests currently being processed, 1 idle workers

CWKCWKWWWWWWWKKWKCKWKWWKKKWKWWRWKWKWKKKWWWCWWKCWKKWWWWKKWKWWWKWW
WWRWWCKKKWKWCKWCWWRWWKWWWWKKCWKWWWWWWKKWKWWWWWKRWWKKWWWWKWK.WWWW
WWKW.WWKWKWKWWKWK.CWWWWKRRKKWKW.WWKCKKKWKCRRWWKWWKWKWWKKKKKWWWWW
WKWCKRKCKWWWCKKRWKWWCKWWWKWRK.WWKKKK.KKKWKWWWWWKW.WKK.KKKKCKKWWW
WWWK.KWKWWKKKKKWKWWK.WW_WKWKWWR.KWKWWWWCWK.WKKKKWWKWKK.KW.KKWKWW
.CW.KWWWWWKRKW.WWKWWW.RRCWWCWKCWKWKKKWKWKKCWKKKWKK.WWC.RWWWCKWWW
KWWKKKKKKWW.KWWKKWKW.KWWCKKWKKWKWKKCWWKWKKWWWWWWWWW.WWKKWKKC.KWW
KKWWKWKKKKWWKKCCKKW.WWKW..WWC.W...KK..KWWWKW.WW.WKW.WKW.....K.KK
..RKKWWW...WW.WKC.WKK.W.KWWWW.KKW..KW...W.KKWWKWKWK.K.W..R......
...W.C..K....K....K..R..WKKWWW..W....KRK..W......W.....WW.W...W.
.K.K.......W.WK...WK.KWW....W....C..K.KKW.W......K...KW.KW..WWW.
.K...W.KKK..WWW.W.WKK.K.W.W.KW.W.W.KK.W.WKKW...K.....WW.W..KK...
.K.KKCW..W.....WKK...KKW.W.WKK..WW....W..W..W...W.........WWWW..
WW.KCWK..W..K.K.W........R.K....................................
................................................................
................................................................
................................................................

at current time the last connection is :
861-0	-	0/0/57	. 	2.74	627	0	0.0	0.00	0.10 	127.0.0.1	server2	OPTIONS * HTTP/1.0

my apache 2.2.16 configuration is :

<IfModule prefork.c>
ServerLimit 1500
MaxClients 1400
MaxRequestsPerChild 150
MinSpareServers 10
MaxSpareServers 20
StartServers 5
KeepAlive On
MaxKeepAliveRequests 51
KeepAliveTimeout 7
TimeOut 70
HostnameLookups Off
ExtendedStatus On
ListenBacklog 100
</IfModule>

and my mysql configuration is :

# The MySQL server
[mysqld]
port            = 3306
socket          = /var/lib/mysql/mysql.sock
skip-external-locking
key_buffer_size = 64M
max_allowed_packet = 1M
table_open_cache = 256
# table_cache = 2048
# table_definition_cache = 1024
long_query_time = 2
sort_buffer_size = 512k
read_buffer_size = 256k
read_rnd_buffer_size = 2M
net_buffer_length = 8K
thread_cache_size = 8
query_cache_size= 32M
query_cache_limit = 16M
query_cache_type= 1

myisam_sort_buffer_size = 16M
wait_timeout = 95
interactive_timeout = 18000
connect_timeout = 70


skip-innodb
skip-blackhole
local-infile=0
slow_query_log=1
slow_query_log_file = /var/lib/mysql/s2-slow-q.log
max_join_size = 16M
#concurrent_insert=2
#max_connections=610
#max_connect_errors = 10
join_buffer_size = 8600
tmp_table_size = 32M
# max_heap_table_size = 32M

# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 16

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
skip-networking

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[myisamchk]
key_buffer_size = 32M
sort_buffer_size = 32M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

Output from vmstat:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
  9  0  84156 100956 179564 6184684    0    0   302   171    2    3 29 20 49  1  1

the php configuration uses lots of disable functions , also suphp , eacclerator , open ssl , mod security

and also server is running with cpanel/whm , with munin and clamd antivirus and csf firewall.

what do you suggest to lower the loads ?
or as you see apache is using about 128 mb ram per connection , how can i lower this ?
thanks in advanced for any suggestions.

[LAZer]

i am going to remove openssl and mod fcgi , fast cgi , because i already used to have performace problems with them and they are not used that much at the sites on my server... maybe removing addtional modules from apache reduces its needed memory.

[cPanelTristan]

EAccelerator will not work with suPHP, so if you are using suPHP, you aren't getting any benefit from OPCode anyway. There's no OPCode that works with suPHP

The only real way to reduce the load quickly and easily is likely to either switch to using DSO or using FCGI. Many people don't like to use DSO because it isn't as secure in many ways (can't easily track down processes as the user because they run as nobody, have to have higher file permissions that aren't as secure). Many people don't like to use FCGI because it requires a lot more configuration out of the box to tweak than suPHP. Both DSO and FCGI run faster and normally will have lower load levels for a highly active machine over suPHP

[LAZer]

thanks for the replies . i saw that eaccelerator is working on the server because it cached many files in its directory but anyway you know better than me. i like suphp too so i dont want to change it to dso or fcgi also i am thinking about changing apache with litespeed.
for example at current time there is 1200 connections on apache and load is around 3 , i dont know exactly why load goes that high some times and i dont know how to use top or htop to find exactly what is causing that high loads . i have found many command about top or ps but i cannot determine what is the exact process that causing that kind of load.
maybe httpd or awstat logs or remoteFTP backup processes or mysql/php processes are the cause of that.
anyway last night i removed fcgi and openssl and some other apache modules and load decreased from 6-7 to 2-3 at the same time ( in comparison with last nights )
this is the errors i recieve these days :

IMPORTANT: Do not ignore this email.
 This is cPanel cpuwatch on server2!
 While processing, the cpu has been
maxed out for more than a 6 hour period.  The current load/uptime line on the server at the time of
this email is
 23:24:50 up 45 days, 13:32,  0 users,  load average: 22.39, 25.01, 32.45
 You should check the server to see why the load is so high and take
steps to lower the load.  If you want stats to continue to run even with a high load; Edit
/var/cpanel/cpanel.config and change extracpus to a number larger then 0 (run
/usr/local/cpanel/startup afterwards to pickup the changes).

also this is the killed commands that despite i have set in csf process tracking ( csf.pignore ) not to kill them they are killed again :

here are the codes i put in csf.pignore :

 
exe:/usr/local/cpanel/bin/cpuwatch
cmd:/usr/local/cpanel/bin/logrunner 8.0 /usr/local/cpanel/3rdparty/bin/awstats.pl
cmd:/usr/local/cpanel/bin/logrunner
exe:/usr/local/cpanel/bin/logrunner
exe:/usr/local/cpanel/3rdparty/bin/awstats.pl
cmd:/usr/local/cpanel/3rdparty/bin/awstats.pl
exe:/usr/bin/perl 
cmd:/usr/bin/perl 
cmd:spamd child
exe:/usr/local/bin/perl
cmd:/usr/local/bin/perl
cmd:/usr/local/bin/perl /usr/local/cpanel/3rdparty/bin/awstats.pl
exe:/usr/local/bin/perl /usr/local/cpanel/3rdparty/bin/awstats.pl

here are the killed commands :

lfd on server2: Excessive resource usage:
Time:         Fri Mar  4 23:36:37 2011 +0330
Account:      xxx
Resource:     Process Time
Exceeded:     3658 > 3600 (seconds)
Executable:   /usr/local/bin/perl
Command Line: cpanellogd - http logs for xxx
PID:          16561
Killed:       Yes


Resource:     Process Time
Exceeded:     3644 > 3600 (seconds)
Executable:   /usr/local/bin/perl
Command Line: cpanellogd - http logs for xxx
PID:          13456
Killed:       Yes

Resource:     Process Time
Exceeded:     3647 > 3600 (seconds)
Executable:   /usr/local/bin/perl
Command Line: /usr/bin/perl /usr/local/cpanel/3rdparty/bin/awstats.pl -config=xxx.com -LogFile=/usr/local/apache/domlogs/xxx.com.bkup -update
PID:          17854
Killed:       Yes

Resource:     Process Time
Exceeded:     3614 > 3600 (seconds)
Executable:   /usr/local/cpanel/bin/cpuwatch
Command Line: /usr/local/cpanel/bin/logrunner 8.0 /usr/local/cpanel/3rdparty/bin/awstats.pl -config=xxx -LogFile=/usr/local/apache/domlogs/xxx.bkup -update
PID:          15026
Killed:       Yes