how do I find the script that cause high load?

[yousef]

Hello,
the server load goes high in less than one minuets to reach 200 , when I run "top" command I noticed that the apache use 40% of the CPU resources and the sleeping tasks is around 300 tasks.

Also in "Main >> Server Status >> CPU/Memory/MySQL Usage" I found that user nobody use a lot of resources.

Also in the service status I found that cpsrvd and ftpd services are failed.

My System info is:
WHM 11.11.0 cPanel 11.16.0-S18450
CENTOS Enterprise 4.6 i686 on standard - WHM X v3.1.0

[claudio]

hi

you can try during this overload see apache status at WHM to see which web site is using this resources...

also you can in tweak settings (at whm either) enable pop flood protection and limit number of times that users can acess their pop

check in your /tmp if there is suspicious files do not run or open then just delete then if you are not sure how check this files

install PRM (process resource monitor) then adjust http, mysql and general configuration rules this program will stop any process that is over a threshold you had determine and will email you about the issue

good luck
claudio

[driverC]

I had the same problem and it turned out that hackers were filling up my customers' guestbooks with spam like crazy maxing out MySQL. Now MySQL always appears on top as sleeping even when it is running which is kind of confusing.

Try this:

1. Log in via SSH.
2. Enter "mysql"
3. Enter "show processlist;"
4. Check if there is any process in the "Query mode" and look how long it has been running. I.e. a process in query mode that was running for like 30 seconds is almost certainly a hacker filling up one of your guestbooks. Look at the username and find the guestbook or phpBB in the user's account and change the directory permissions to CHMOD 000.

Another reason why my servers were overloaded recently was a large amount of spam being sent to my server. I could greatly improve the performance by enabling anti spam protection on MTA level: WHM -> Exim configuration and enable Spamcop blacklist.

Also look at the Apache status page in WHM and look at the Apache processes that are running at the moment. Maybe someone is using a script that is causing a lot of load. Or if you see a lot of ? signs on the status page someone may be doing a DDoS against your server.

[yousef]

thanks for help guys
Love you

[erick_paper]

I could greatly improve the performance by enabling anti spam protection on MTA level: WHM -> Exim configuration and enable Spamcop blacklist.

I decided not to use spamcop because it really has a lot of false positives.

Also look at the Apache status page in WHM and look at the Apache processes that are running at the moment. Maybe someone is using a script that is causing a lot of load. Or if you see a lot of ? signs on the status page someone may be doing a DDoS against your server.

This sounds useful. How should I know which IP is calling what? Or which script? The output of "Apache status" seems very geeky, with lots of "......" (dots).

[driverC]

This sounds useful. How should I know which IP is calling what? Or which script? The output of "Apache status" seems very geeky, with lots of "......" (dots).

When logged in via shell enter "top". Then press the "i" key to look at running processes only. You may see some Apache processes running for a long amount of time. Now quickly reload the Apache status page and try to find the process ID in the list...this way you will know which site/domain causes the high load. Usually proxy or chat scripts can cause issues. If you see only dots and question marks it is most likely a DDOS attack. In this case edit your httpd.conf and reduce the timeout:

pico /usr/local/apache/conf/httpd.conf

Change:
Timeout 300

to:
Timeout 15

This usually solves DDOS attacks against Apache but in some cases you will need to do more than this. Also if you like paste a copy of these commands here and I'll let you know what I can see:

top -c -b -n 1

mysql
show processlist;

service httpd fullstatus

(the fullstatus command may not work...in this case copy the Apache server status page from WHM)

[erick_paper]

Thanks so much for this really useful and helpful post. I have installed Apache 2.2.6, which seems to be working faster.

Firstly, I have installed dos_evasive and tuned the timeout and MaxClients stuff. Here are my settings from httpd.conf:

Timeout 15
KeepAlive On
MaxKeepAliveRequests 125
KeepAliveTimeout 2
MinSpareServers 16
MaxSpareServers 32
StartServers 5
MaxClients 250
MaxRequestsPerChild 500

LoadModule security2_module modules/mod_security2.so
LoadModule auth_passthrough_module modules/mod_auth_passthrough.so
LoadModule bwlimited_module modules/mod_bwlimited.so

#---- EVASIVE MODULE
LoadModule evasive20_module modules/mod_evasive20.so
DOSHashTableSize 3097
DOSPageCount 2
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 100
DOSLogDir "/var/log/apache/mod_evasive"

#---- DEFLATE MODULE
LoadModule deflate_module modules/mod_deflate.so
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/x-httpd-php application/x-javascript
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
SetOutputFilter DEFLATE
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.pdf$ no-gzip dont-vary
Header append Vary User-Agent env=!dont-vary

Secondly, I have also installed the "(D)Dos Deflate" thing from medialayer: http://deflate.medialayer.com/

Thirdly, I have installed Chirpy's CSF/LFD and enabled people who connect more than 150 times per minute to be blocked, plus I have enabled the new "Syn Flood" feature with a "5/s" timing thing. Is there a better tool for blocking Syn Floods?

Fourthly, I have also installed "atop" instead of top, and I can tell now which processes are taking the most amount of memory and resources. PLus there is that netstat command to see how many TCP/Syn connections are active.

Fifthly, I have installed the new mod_sec2, and installed many rules from gotroot.com -- including blacklisting of common compromised boxes, common rootkit attacks, common spambots etc.

The server seems to be much better now. Apart from these precautions, anything else you can think of that I can use to *prevent* DDoS attacks?

So many thanks for the info!

[Kent Brockman]

Fourthly, I have also installed "atop" instead of top, and I can tell now which processes are taking the most amount of memory and resources. PLus there is that netstat command to see how many TCP/Syn connections are active.

Hi! where can i get that "atop"?
Cheerz

[troxalias]

You can find here (http://www.cphelp.gr/kb/?View=entry&EntryID=14) a perl script that automatically monitors mysql and kills users which exceed a specific number of connections or Select Queries that run for a long time. It may be helpful in case you have many dynamic sites on your server.

[Kent Brockman]

You can find here (http://www.cphelp.gr/kb/?View=entry&EntryID=14) a perl script that automatically monitors mysql and kills users which exceed a specific number of connections or Select Queries that run for a long time. It may be helpful in case you have many dynamic sites on your server.

Cool. I see the code and this script can autoinstall itself to run every 60 seconds. Before starting it: how much resources it consumes? and how to stop it in case I don't finally need it? can I simply and easily kill the pid and exclude its entry from the crontab?

[troxalias]

Well in my system it uses around 4MB of RAM in idle mode. While running (every 60 seconds by default) it depends on how large is the output of "show processlist" command of mysql and the connection by itself. I could say that in no way it would require more than 20 MB of RAM. To stop it, just kill the processes pid. There are no entries in the crontab for this to work, but if you want to start in on server reboot you could add it to your rc.sysinit (depending on your distro).

[Kent Brockman]

Well in my system it uses around 4MB of RAM in idle mode. While running (every 60 seconds by default) it depends on how large is the output of "show processlist" command of mysql and the connection by itself. I could say that in no way it would require more than 20 MB of RAM. To stop it, just kill the processes pid. There are no entries in the crontab for this to work, but if you want to start in on server reboot you could add it to your rc.sysinit (depending on your distro).

ok, i'll try to give it a try when posible.
best regards!

[Kent Brockman]

Also look at the Apache status page in WHM and look at the Apache processes that are running at the moment. Maybe someone is using a script that is causing a lot of load. Or if you see a lot of ? signs on the status page someone may be doing a DDoS against your server.

hi, in case I find those ? signs on the status page, how can be stopped that DDoS attack?