The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server Slow - Extremely High CPU Usage

Discussion in 'General Discussion' started by ralbano, Nov 27, 2008.

  1. ralbano

    ralbano Member

    Joined:
    Mar 9, 2008
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    Hello, I have an issue, not the first time it happens.
    The server start responding very slow (all services), logged in I run an uptime and get really HIGH cpu usage values, like this:

    [root@cp03 ~]# uptime
    18:38:27 up 33 days, 22:42, 1 user, load average: 86.75, 81.58, 61.51

    growing up minute a minute...

    As soon as I see this, I run a top to find the preccess causing this high CPU usage, and for sorprise I can't see any process consuming lot of CPU and the CPU usage from top is normal:

    top - 18:38:25 up 33 days, 22:42, 1 user, load average: 86.75, 81.58, 61.51
    Tasks: 550 total, 1 running, 546 sleeping, 1 stopped, 2 zombie
    Cpu(s): 9.6%us, 1.5%sy, 0.0%ni, 0.0%id, 88.6%wa, 0.0%hi, 0.3%si, 0.0%st
    Mem: 3097584k total, 2579480k used, 518104k free, 61120k buffers
    Swap: 4192924k total, 2709180k used, 1483744k free, 415360k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    10073 nobody 20 0 62096 27m 2112 D 9 0.9 0:01.92 httpd
    15079 nobody 20 0 54612 20m 2148 D 3 0.7 0:01.73 httpd
    10009 nobody 20 0 56248 19m 2292 D 2 0.7 0:03.97 httpd
    18802 root 20 0 2668 1308 780 R 1 0.0 0:01.87 top
    20287 nobody 20 0 45668 10m 1688 D 1 0.4 0:00.02 httpd
    20395 nobody 20 0 47964 13m 1748 D 1 0.4 0:00.07 httpd
    23111 root 20 0 5868 2844 1364 S 1 0.1 6:39.95 authProg
    3374 mailman 20 0 60144 2908 2408 S 0 0.1 9:15.69 python2.4
    3375 mailman 20 0 1847m 200m 2380 D 0 6.6 26:12.36 python2.4
    3378 mailman 20 0 10896 2344 1956 S 0 0.1 7:20.07 python2.4
    3379 mailman 20 0 98792 28m 2540 D 0 0.9 97:00.47 python2.4
    15185 nobody 20 0 54096 19m 2152 S 0 0.6 0:01.04 httpd
    15610 nobody 20 0 61168 26m 2084 S 0 0.9 0:01.84 httpd
    15801 nobody 20 0 54264 19m 1848 S 0 0.6 0:01.85 httpd
    18989 mailnull 20 0 11408 4088 2836 S 0 0.1 0:00.09 exim

    I'm really desoriented with this issue, any clue???
     
  2. stdout

    stdout Well-Known Member

    Joined:
    Apr 10, 2003
    Messages:
    189
    Likes Received:
    5
    Trophy Points:
    18
    Location:
    Nelspruit, Mpumalanga, South Africa
    cPanel Access Level:
    Root Administrator
    One thing is clear - you have alot of running processes.
    Is it a busy website? Is it a bad cron? Perhaps a bad script?

    Are you running suPHP?
    1. Determine who is runinng the most processes and troubleshoot from there.
    Code:
    ps aux | awk {'print $1'} | sort | uniq -c | sort -n
    PS. This looks a bit sketchy - may be worth looking into:
     
  3. hzJayJ

    hzJayJ Well-Known Member

    Joined:
    Nov 14, 2008
    Messages:
    76
    Likes Received:
    1
    Trophy Points:
    8
    Have you enabled user nobody to send mails in the server?

    Also check whether there is any spamming going on in the server

    You can use the below scritp to check whether nobody spamming is going on in the server

    awk '{ if ($0 ~ "cwd" && $0 ~ "home") {print $3} }' /var/log/exim_mainlog | sort | uniq -c | sort -nk 1

    For this you need to enable extended login in the server.
    To enable extened login

    1. Open exim.conf
    pico /etc/exim.conf

    2) Find this;
    Ctrl + W: hostlist auth_relay_hosts = *

    #########################
    Runtime configuration file for Exim #
    #########################



    3) After hostlist auth_relay_hosts = *

    add the following

    log_selector = +address_rewrite +all_parents +arguments +connection_reject +delay_delivery +delivery_size +dnslist_defer +incoming_interface +incoming_port +lost_incoming_connection +queue_run +received_sender +received_recipients +retry_defer +sender_on_delivery +size_reject +skip_delivery +smtp_confirmation +smtp_connection +smtp_protocol_error +smtp_syntax_error +subject +tls_cipher +tls_peerdn


    Also check for the mysql process running in the server, which might be the cause for high load, you can use the command mysqladminproc to list the mysql process. If mysql is using the load try restarting the mysql and check whether the load comes down.
     
  4. Ghulam Yaseen

    Ghulam Yaseen Well-Known Member

    Joined:
    Mar 14, 2008
    Messages:
    49
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Karachi,Pakistan
    linux/ Unix thirst

    Hello :),

    I believe to kill apache and exim and restart them after the load lowers:cool:


     
  5. smartads

    smartads Member

    Joined:
    Dec 19, 2008
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    You should disable "nobody" from being used from your server since it's a security issue. Your server may be relaying email as a spam network. I would double check to see if your server has been flagged as spammy within spam databases online.

    Make sure to use "capcha security images" for all of your email forms and always use a valid email recipent. Double check and verify each users email for valid formatting.

    If you use "tell a friend scripts", get rid of them since good hackers can use the BCC to exploit your email script to send millions of emails in no time flat. (Happened to me years ago, not fun)

    Cheers!

    Martin Lemieux
     
  6. ralbano

    ralbano Member

    Joined:
    Mar 9, 2008
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    nobody bloqued form sendind emails

    Yes, It seems that blocking nobody user fom sending emails make the difference.
    I belive there is a lot of buggy php pages in my server that are fresh meat to spammers.
    Take a look at the following graphics:

    [​IMG]

    Bye!.
     
  7. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    Switch to using suphp and all user PHP processes will run as the user rather than nobody.
    That's a good thing to check. In this case though, it's unlikely you're "relaying" email in the normal sense of the word "relay", although you may have loose scripts generating spam.

    The best way to kill these scripts off is to use mod_security with a ruleset that catches Bcc, Cc and To in submitted fields.

    One easy way to check the format is to write a small antispam script which checks all input fields and include it in your user's vulnerable scripts. The nice thing about doing this is that it's a one-line modification to fix a poor script. However, mod_security will catch and block most of this.

    The problem is actually not with "tell a friend" scripts as much as it is with poorly secured form scripts of any sort. It's simply a matter of making sure the scripts are secure, with suphp/mod_security and some antispammer field checks in the scripts. A "tell a friend" script can be completely secure; if done well, you need never, ever, suffer spam. The use of captcha or recaptcha scripts nails form spam very nicely.

    The other useful thing is using the cpanel outgoing email limit to catch any users that start generating spam. Works like a charm ...

    If you couple mod_security with CSF, the spammer IP will get blocked after they've tried a few times - a very, very useful feature as it makes it MUCH harder for them to try other exploits. (CSF = Configserver Firewall = http://www.configserver.com/cp/csf.html).

    Ralbano - these tips will help you also. The solution is to build in sensible safeguards at a number of levels - a single level of safeguard can always get bypassed. If you combine suphp, mod_security with a good set of rules, CSF for blocking baddies, outgoing email limits in cpanel, use of a PHP antispam script where appropriate, and either SMTP tweak or CSF port 25 blocking, there's not much that can get past.
     
    #7 brianoz, Dec 31, 2008
    Last edited: Dec 31, 2008
  8. ralbano

    ralbano Member

    Joined:
    Mar 9, 2008
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    I agree.
    currently I'm using CFG to detect abusive relaying (local or remote), then investigating manually.
    Also I have some scripts that help me in detecting abnormal queue issues (for example, lot of emails with same subject)
    And alse I keep an eye on how queue grows with some automated graphics generation.
     
Loading...

Share This Page