The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

high load problem with high traffic on apache at nights

Discussion in 'EasyApache' started by LAZer, Mar 4, 2011.

  1. LAZer

    LAZer Well-Known Member

    Joined:
    Jan 18, 2010
    Messages:
    70
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    at net :D
    hi , i recently faced a problem with my server load averages . sites traffic on my server increased recently more than 200000 page views daily is on some of the sites on my server . and i receive loads around 25.

    every thing ( page load/download speed ) still are working normal with this high load because the server is a great one but the log softwares send me alerts about the high cpuload problem having a high load for more than 6-7 hours.

    my server ram is 8 gb ddr II and it has 2* cuad core xeon cpus, the hards are raid10 and io rates are normal . the 2 things that are causing high load are apache connections and mysql connections which i optimized them the best i could . and that configuration was working fine for more than 4 5 monthes.

    here is a sample of my top :

    Code:
    root@server2 [~]# w
     18:53:11 up 45 days,  9:00,  1 user,  load average: 18.19, 16.73, 15.50
    at high load times there are lots of processes like this :
    Code:
    13182 nobody    15   0  125m  28m 3092 S  1.0  0.3   0:00.64 httpd
    25322 nobody    15   0  125m  28m 3100 S  1.0  0.3   0:00.88 httpd
     1300 nobody    15   0  125m  28m 3092 S  0.7  0.3   0:01.33 httpd
    
    nobody   22619  0.1  0.3 128612 28748 ?        S    18:59   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   22620  0.1  0.3 128612 28680 ?        S    18:59   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    
     4203 xxx  16   0     0    0    0 Z 24.1  0.0   0:00.16 php <defunct>
     4205 xxx  16   0  131m  15m 9516 R 21.1  0.2   0:00.14 php
     4197 xxx  17   0  131m  15m 9516 R 15.1  0.2   0:00.10 php
     4204 xxx   16   0  130m  15m  10m S 12.1  0.2   0:00.08 php
    
    xxx is the user name
    
    mysql    11959 53.1  1.1 342080 98968 ?        Sl   18:28  20:33 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --log-error=/var/lib/mysql/server2..err --pid-file=/var/lib/mysql/server2.pid --socket=/var/lib/mysql/mysql.sock --port=3306
    
    
    there is a little io waiting and memory swaping , also memory leaks are very low :

    Code:
    root@server2 [~]# iostat
    Linux 2.6.18-194.32.1.el5xen (server2)       03/04/2011
    
    [B]avg-cpu:  %user   %nice %system [COLOR="#00ff00"]%iowait[/COLOR]  %steal   %idle
              28.24    0.56   19.72    [COLOR="Lime"]1.31[/COLOR]    1.37   48.78[/B]
    
    Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
    xvdb             39.60      2886.19        97.86 3834298211  130001032
    xvdb1            39.60      2886.19        97.86 3834297763  130001032
    xvda            162.98      1094.28      2236.21 1453748336 2970803498
    xvda1             0.00         0.00         0.00       2724         98
    xvda2           162.98      1094.28      2236.21 1453745412 2970803400
    xvdc             25.56       850.02       319.08 1129245211  423897888
    xvdc1            25.56       850.02       319.08 1129244763  423897888
    dm-0            307.20      1094.28      2236.04 1453744330 2970576464
    dm-1              0.02         0.00         0.17        736     226936
    dm-2             54.91       850.02       319.08 1129244194  423897888
    dm-3             52.31      2886.19        97.86 3834297322  130001032
    Code:
    root@server2 [~]# free -m
                 total       used       free     shared    buffers     cached
    Mem:          8192       7969        222          0        176       6109
    -/+ buffers/cache:       1683       6508
    Swap:         5087         82       5005
    Code:
    Apache Server Status for localhost
    
      [B] Server Version: Apache/2.2.16 (Unix) mod_ssl/2.2.16
              OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4
              Perl/v5.8.8[/B]
    
       Server Built: Aug 15 2010 09:50:04
         _________________________________________________________________
    
       Current Time: Friday, 04-Mar-2011 19:12:50 IRST
       Restart Time: Friday, 04-Mar-2011 18:25:41 IRST
       Parent Server Generation: 0
       Server uptime: 47 minutes 9 seconds
       Total accesses: 346214 - Total Traffic: 307.0 GB
       CPU Usage: u384.42 s157.16 cu2403.91 cs0 - [COLOR="#ff0000"]134% CPU load[/COLOR]
       122 requests/sec - 111.1 MB/second - 0.9 MB/request
       624 requests currently being processed, 1 idle workers
    
    CWKCWKWWWWWWWKKWKCKWKWWKKKWKWWRWKWKWKKKWWWCWWKCWKKWWWWKKWKWWWKWW
    WWRWWCKKKWKWCKWCWWRWWKWWWWKKCWKWWWWWWKKWKWWWWWKRWWKKWWWWKWK.WWWW
    WWKW.WWKWKWKWWKWK.CWWWWKRRKKWKW.WWKCKKKWKCRRWWKWWKWKWWKKKKKWWWWW
    WKWCKRKCKWWWCKKRWKWWCKWWWKWRK.WWKKKK.KKKWKWWWWWKW.WKK.KKKKCKKWWW
    WWWK.KWKWWKKKKKWKWWK.WW_WKWKWWR.KWKWWWWCWK.WKKKKWWKWKK.KW.KKWKWW
    .CW.KWWWWWKRKW.WWKWWW.RRCWWCWKCWKWKKKWKWKKCWKKKWKK.WWC.RWWWCKWWW
    KWWKKKKKKWW.KWWKKWKW.KWWCKKWKKWKWKKCWWKWKKWWWWWWWWW.WWKKWKKC.KWW
    KKWWKWKKKKWWKKCCKKW.WWKW..WWC.W...KK..KWWWKW.WW.WKW.WKW.....K.KK
    ..RKKWWW...WW.WKC.WKK.W.KWWWW.KKW..KW...W.KKWWKWKWK.K.W..R......
    ...W.C..K....K....K..R..WKKWWW..W....KRK..W......W.....WW.W...W.
    .K.K.......W.WK...WK.KWW....W....C..K.KKW.W......K...KW.KW..WWW.
    .K...W.KKK..WWW.W.WKK.K.W.W.KW.W.W.KK.W.WKKW...K.....WW.W..KK...
    .K.KKCW..W.....WKK...KKW.W.WKK..WW....W..W..W...W.........WWWW..
    WW.KCWK..W..K.K.W........R.K....................................
    ................................................................
    ................................................................
    ................................................................
    
    at current time the last connection is :
    [COLOR="Red"]861-[/COLOR]0	-	0/0/57	. 	2.74	627	0	0.0	0.00	0.10 	127.0.0.1	server2	OPTIONS * HTTP/1.0
    my apache 2.2.16 configuration is :
    Code:
    <IfModule prefork.c>
    ServerLimit 1500
    MaxClients 1400
    MaxRequestsPerChild 150
    MinSpareServers 10
    MaxSpareServers 20
    StartServers 5
    KeepAlive On
    MaxKeepAliveRequests 51
    KeepAliveTimeout 7
    TimeOut 70
    HostnameLookups Off
    ExtendedStatus On
    ListenBacklog 100
    </IfModule>
    and my mysql configuration is :
    Code:
    # The MySQL server
    [mysqld]
    port            = 3306
    socket          = /var/lib/mysql/mysql.sock
    skip-external-locking
    key_buffer_size = 64M
    max_allowed_packet = 1M
    table_open_cache = 256
    # table_cache = 2048
    # table_definition_cache = 1024
    long_query_time = 2
    sort_buffer_size = 512k
    read_buffer_size = 256k
    read_rnd_buffer_size = 2M
    net_buffer_length = 8K
    thread_cache_size = 8
    query_cache_size= 32M
    query_cache_limit = 16M
    query_cache_type= 1
    
    myisam_sort_buffer_size = 16M
    wait_timeout = 95
    interactive_timeout = 18000
    connect_timeout = 70
    
    
    skip-innodb
    skip-blackhole
    local-infile=0
    slow_query_log=1
    slow_query_log_file = /var/lib/mysql/s2-slow-q.log
    max_join_size = 16M
    #concurrent_insert=2
    #max_connections=610
    #max_connect_errors = 10
    join_buffer_size = 8600
    tmp_table_size = 32M
    # max_heap_table_size = 32M
    
    # Try number of CPU's*2 for thread_concurrency
    thread_concurrency = 16
    
    # Don't listen on a TCP/IP port at all. This can be a security enhancement,
    # if all processes that need to connect to mysqld run on the same host.
    # All interaction with mysqld must be made via Unix sockets or named pipes.
    # Note that using this option without enabling named pipes on Windows
    # (via the "enable-named-pipe" option) will render mysqld useless!
    #
    skip-networking
    
    [mysqldump]
    quick
    max_allowed_packet = 16M
    
    [mysql]
    no-auto-rehash
    # Remove the next comment character if you are not familiar with SQL
    #safe-updates
    
    [myisamchk]
    key_buffer_size = 32M
    sort_buffer_size = 32M
    read_buffer = 2M
    write_buffer = 2M
    
    [mysqlhotcopy]
    interactive-timeout
    Code:
    Output from vmstat:
    procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
      r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
      9  0  84156 100956 179564 6184684    0    0   302   171    2    3 29 20 49  1  1
    the php configuration uses lots of disable functions , also suphp , eacclerator , open ssl , mod security

    and also server is running with cpanel/whm , with munin and clamd antivirus and csf firewall.

    what do you suggest to lower the loads ?
    or as you see apache is using about 128 mb ram per connection , how can i lower this ?
    thanks in advanced for any suggestions.:confused::confused:
     
    #1 LAZer, Mar 4, 2011
    Last edited: Mar 4, 2011
  2. LAZer

    LAZer Well-Known Member

    Joined:
    Jan 18, 2010
    Messages:
    70
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    at net :D
    i am going to remove openssl and mod fcgi , fast cgi , because i already used to have performace problems with them and they are not used that much at the sites on my server... maybe removing addtional modules from apache reduces its needed memory.
     
  3. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    EAccelerator will not work with suPHP, so if you are using suPHP, you aren't getting any benefit from OPCode anyway. There's no OPCode that works with suPHP

    The only real way to reduce the load quickly and easily is likely to either switch to using DSO or using FCGI. Many people don't like to use DSO because it isn't as secure in many ways (can't easily track down processes as the user because they run as nobody, have to have higher file permissions that aren't as secure). Many people don't like to use FCGI because it requires a lot more configuration out of the box to tweak than suPHP. Both DSO and FCGI run faster and normally will have lower load levels for a highly active machine over suPHP
     
  4. LAZer

    LAZer Well-Known Member

    Joined:
    Jan 18, 2010
    Messages:
    70
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    at net :D
    thanks for the replies . i saw that eaccelerator is working on the server because it cached many files in its directory but anyway you know better than me. i like suphp too so i dont want to change it to dso or fcgi :( also i am thinking about changing apache with litespeed.
    for example at current time there is 1200 connections on apache and load is around 3 , i dont know exactly why load goes that high some times :( and i dont know how to use top or htop to find exactly what is causing that high loads . i have found many command about top or ps but i cannot determine what is the exact process that causing that kind of load.
    maybe httpd or awstat logs or remoteFTP backup processes or mysql/php processes are the cause of that.
    anyway last night i removed fcgi and openssl and some other apache modules and load decreased from 6-7 to 2-3 at the same time ( in comparison with last nights )
    this is the errors i recieve these days :
    Code:
    IMPORTANT: Do not ignore this email.
     This is cPanel cpuwatch on server2!
     While processing, the cpu has been
    maxed out for more than a 6 hour period.  The current load/uptime line on the server at the time of
    this email is
     23:24:50 up 45 days, 13:32,  0 users,  load average: 22.39, 25.01, 32.45
     You should check the server to see why the load is so high and take
    steps to lower the load.  If you want stats to continue to run even with a high load; Edit
    /var/cpanel/cpanel.config and change extracpus to a number larger then 0 (run
    /usr/local/cpanel/startup afterwards to pickup the changes).
    
    also this is the killed commands that despite i have set in csf process tracking ( csf.pignore ) not to kill them they are killed again :

    here are the codes i put in csf.pignore :
    Code:
     
    exe:/usr/local/cpanel/bin/cpuwatch
    cmd:/usr/local/cpanel/bin/logrunner 8.0 /usr/local/cpanel/3rdparty/bin/awstats.pl
    cmd:/usr/local/cpanel/bin/logrunner
    exe:/usr/local/cpanel/bin/logrunner
    exe:/usr/local/cpanel/3rdparty/bin/awstats.pl
    cmd:/usr/local/cpanel/3rdparty/bin/awstats.pl
    exe:/usr/bin/perl 
    cmd:/usr/bin/perl 
    cmd:spamd child
    exe:/usr/local/bin/perl
    cmd:/usr/local/bin/perl
    cmd:/usr/local/bin/perl /usr/local/cpanel/3rdparty/bin/awstats.pl
    exe:/usr/local/bin/perl /usr/local/cpanel/3rdparty/bin/awstats.pl
    here are the killed commands :
    Code:
    lfd on server2: Excessive resource usage:
    Time:         Fri Mar  4 23:36:37 2011 +0330
    Account:      xxx
    Resource:     Process Time
    Exceeded:     3658 > 3600 (seconds)
    Executable:   /usr/local/bin/perl
    Command Line: cpanellogd - http logs for xxx
    PID:          16561
    Killed:       Yes
    
    
    Resource:     Process Time
    Exceeded:     3644 > 3600 (seconds)
    Executable:   /usr/local/bin/perl
    Command Line: cpanellogd - http logs for xxx
    PID:          13456
    Killed:       Yes
    
    Resource:     Process Time
    Exceeded:     3647 > 3600 (seconds)
    Executable:   /usr/local/bin/perl
    Command Line: /usr/bin/perl /usr/local/cpanel/3rdparty/bin/awstats.pl -config=xxx.com -LogFile=/usr/local/apache/domlogs/xxx.com.bkup -update
    PID:          17854
    Killed:       Yes
    
    Resource:     Process Time
    Exceeded:     3614 > 3600 (seconds)
    Executable:   /usr/local/cpanel/bin/cpuwatch
    Command Line: /usr/local/cpanel/bin/logrunner 8.0 /usr/local/cpanel/3rdparty/bin/awstats.pl -config=xxx -LogFile=/usr/local/apache/domlogs/xxx.bkup -update
    PID:          15026
    Killed:       Yes
     
Loading...

Share This Page