Server Slow - Extremely High CPU Usage

[ralbano]

Hello, I have an issue, not the first time it happens.
The server start responding very slow (all services), logged in I run an uptime and get really HIGH cpu usage values, like this:

[[email protected] ~]# uptime
18:38:27 up 33 days, 22:42, 1 user, load average: 86.75, 81.58, 61.51

growing up minute a minute...

As soon as I see this, I run a top to find the preccess causing this high CPU usage, and for sorprise I can't see any process consuming lot of CPU and the CPU usage from top is normal:

top - 18:38:25 up 33 days, 22:42, 1 user, load average: 86.75, 81.58, 61.51
Tasks: 550 total, 1 running, 546 sleeping, 1 stopped, 2 zombie
Cpu(s): 9.6%us, 1.5%sy, 0.0%ni, 0.0%id, 88.6%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 3097584k total, 2579480k used, 518104k free, 61120k buffers
Swap: 4192924k total, 2709180k used, 1483744k free, 415360k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10073 nobody 20 0 62096 27m 2112 D 9 0.9 0:01.92 httpd
15079 nobody 20 0 54612 20m 2148 D 3 0.7 0:01.73 httpd
10009 nobody 20 0 56248 19m 2292 D 2 0.7 0:03.97 httpd
18802 root 20 0 2668 1308 780 R 1 0.0 0:01.87 top
20287 nobody 20 0 45668 10m 1688 D 1 0.4 0:00.02 httpd
20395 nobody 20 0 47964 13m 1748 D 1 0.4 0:00.07 httpd
23111 root 20 0 5868 2844 1364 S 1 0.1 6:39.95 authProg
3374 mailman 20 0 60144 2908 2408 S 0 0.1 9:15.69 python2.4
3375 mailman 20 0 1847m 200m 2380 D 0 6.6 26:12.36 python2.4
3378 mailman 20 0 10896 2344 1956 S 0 0.1 7:20.07 python2.4
3379 mailman 20 0 98792 28m 2540 D 0 0.9 97:00.47 python2.4
15185 nobody 20 0 54096 19m 2152 S 0 0.6 0:01.04 httpd
15610 nobody 20 0 61168 26m 2084 S 0 0.9 0:01.84 httpd
15801 nobody 20 0 54264 19m 1848 S 0 0.6 0:01.85 httpd
18989 mailnull 20 0 11408 4088 2836 S 0 0.1 0:00.09 exim

I'm really desoriented with this issue, any clue???

 

[stdout]

One thing is clear - you have alot of running processes.
Is it a busy website? Is it a bad cron? Perhaps a bad script?

Are you running suPHP?
1. Determine who is runinng the most processes and troubleshoot from there.

ps aux | awk {'print $1'} | sort | uniq -c | sort -n

PS. This looks a bit sketchy - may be worth looking into:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3375 mailman 20 0 1847m 200m 2380 D 0 6.6 26:12.36 python2.4

 

[hzJayJ]

Have you enabled user nobody to send mails in the server?

Also check whether there is any spamming going on in the server

You can use the below scritp to check whether nobody spamming is going on in the server

awk '{ if ($0 ~ "cwd" && $0 ~ "home") {print $3} }' /var/log/exim_mainlog | sort | uniq -c | sort -nk 1

For this you need to enable extended login in the server.
To enable extened login

1. Open exim.conf
pico /etc/exim.conf

2) Find this;
Ctrl + W: hostlist auth_relay_hosts = *

#########################
Runtime configuration file for Exim #
#########################



3) After hostlist auth_relay_hosts = *

add the following

log_selector = +address_rewrite +all_parents +arguments +connection_reject +delay_delivery +delivery_size +dnslist_defer +incoming_interface +incoming_port +lost_incoming_connection +queue_run +received_sender +received_recipients +retry_defer +sender_on_delivery +size_reject +skip_delivery +smtp_confirmation +smtp_connection +smtp_protocol_error +smtp_syntax_error +subject +tls_cipher +tls_peerdn


Also check for the mysql process running in the server, which might be the cause for high load, you can use the command mysqladminproc to list the mysql process. If mysql is using the load try restarting the mysql and check whether the load comes down.

 

[Ghulam Yaseen]

linux/ Unix thirst

Hello ,

I believe to kill apache and exim and restart them after the load lowers

Hello, I have an issue, not the first time it happens.
The server start responding very slow (all services), logged in I run an uptime and get really HIGH cpu usage values, like this:

[[email protected] ~]# uptime
18:38:27 up 33 days, 22:42, 1 user, load average: 86.75, 81.58, 61.51

growing up minute a minute...

As soon as I see this, I run a top to find the preccess causing this high CPU usage, and for sorprise I can't see any process consuming lot of CPU and the CPU usage from top is normal:

top - 18:38:25 up 33 days, 22:42, 1 user, load average: 86.75, 81.58, 61.51
Tasks: 550 total, 1 running, 546 sleeping, 1 stopped, 2 zombie
Cpu(s): 9.6%us, 1.5%sy, 0.0%ni, 0.0%id, 88.6%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 3097584k total, 2579480k used, 518104k free, 61120k buffers
Swap: 4192924k total, 2709180k used, 1483744k free, 415360k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10073 nobody 20 0 62096 27m 2112 D 9 0.9 0:01.92 httpd
15079 nobody 20 0 54612 20m 2148 D 3 0.7 0:01.73 httpd
10009 nobody 20 0 56248 19m 2292 D 2 0.7 0:03.97 httpd
18802 root 20 0 2668 1308 780 R 1 0.0 0:01.87 top
20287 nobody 20 0 45668 10m 1688 D 1 0.4 0:00.02 httpd
20395 nobody 20 0 47964 13m 1748 D 1 0.4 0:00.07 httpd
23111 root 20 0 5868 2844 1364 S 1 0.1 6:39.95 authProg
3374 mailman 20 0 60144 2908 2408 S 0 0.1 9:15.69 python2.4
3375 mailman 20 0 1847m 200m 2380 D 0 6.6 26:12.36 python2.4
3378 mailman 20 0 10896 2344 1956 S 0 0.1 7:20.07 python2.4
3379 mailman 20 0 98792 28m 2540 D 0 0.9 97:00.47 python2.4
15185 nobody 20 0 54096 19m 2152 S 0 0.6 0:01.04 httpd
15610 nobody 20 0 61168 26m 2084 S 0 0.9 0:01.84 httpd
15801 nobody 20 0 54264 19m 1848 S 0 0.6 0:01.85 httpd
18989 mailnull 20 0 11408 4088 2836 S 0 0.1 0:00.09 exim

I'm really desoriented with this issue, any clue???

 

[smartads]

You should disable "nobody" from being used from your server since it's a security issue. Your server may be relaying email as a spam network. I would double check to see if your server has been flagged as spammy within spam databases online.

Make sure to use "capcha security images" for all of your email forms and always use a valid email recipent. Double check and verify each users email for valid formatting.

If you use "tell a friend scripts", get rid of them since good hackers can use the BCC to exploit your email script to send millions of emails in no time flat. (Happened to me years ago, not fun)

Cheers!

Martin Lemieux

 

[ralbano]

nobody bloqued form sendind emails

Yes, It seems that blocking nobody user fom sending emails make the difference.
I belive there is a lot of buggy php pages in my server that are fresh meat to spammers.
Take a look at the following graphics:

Bye!.

 

[brianoz]

You should disable "nobody" from being used from your server since it's a security issue.

Switch to using suphp and all user PHP processes will run as the user rather than nobody.

Your server may be relaying email as a spam network. I would double check to see if your server has been flagged as spammy within spam databases online.

That's a good thing to check. In this case though, it's unlikely you're "relaying" email in the normal sense of the word "relay", although you may have loose scripts generating spam.

The best way to kill these scripts off is to use mod_security with a ruleset that catches Bcc, Cc and To in submitted fields.

Make sure to use "capcha security images" for all of your email forms and always use a valid email recipent. Double check and verify each users email for valid formatting.

One easy way to check the format is to write a small antispam script which checks all input fields and include it in your user's vulnerable scripts. The nice thing about doing this is that it's a one-line modification to fix a poor script. However, mod_security will catch and block most of this.

If you use "tell a friend scripts", get rid of them since good hackers can use the BCC to exploit your email script to send millions of emails in no time flat. (Happened to me years ago, not fun)

The problem is actually not with "tell a friend" scripts as much as it is with poorly secured form scripts of any sort. It's simply a matter of making sure the scripts are secure, with suphp/mod_security and some antispammer field checks in the scripts. A "tell a friend" script can be completely secure; if done well, you need never, ever, suffer spam. The use of captcha or recaptcha scripts nails form spam very nicely.

The other useful thing is using the cpanel outgoing email limit to catch any users that start generating spam. Works like a charm ...

If you couple mod_security with CSF, the spammer IP will get blocked after they've tried a few times - a very, very useful feature as it makes it MUCH harder for them to try other exploits. (CSF = Configserver Firewall = http://www.configserver.com/cp/csf.html).

Ralbano - these tips will help you also. The solution is to build in sensible safeguards at a number of levels - a single level of safeguard can always get bypassed. If you combine suphp, mod_security with a good set of rules, CSF for blocking baddies, outgoing email limits in cpanel, use of a PHP antispam script where appropriate, and either SMTP tweak or CSF port 25 blocking, there's not much that can get past.

 

[ralbano]

I agree.
currently I'm using CFG to detect abusive relaying (local or remote), then investigating manually.
Also I have some scripts that help me in detecting abnormal queue issues (for example, lot of emails with same subject)
And alse I keep an eye on how queue grows with some automated graphics generation.