server load go to maximum

[Stanleytiew]

I receive an email from root as follow:

IMPORTANT: Do not ignore this email.
This is cPanel stats runner on ns1.xxxxxxx.com!
While processing the log files for user xxxx, the cpu has been maxed out for more then a 6 hour period. The current load/uptime line on the server at the time of this email is
07:41:52 up 11:28, 0 users, load average: 5.00, 5.00, 5.00
You should check the server to see why the load is so high and take steps to lower the load. If you want stats to continue to run even with a high load; Edit /var/cpanel/cpanel.config and change extracpus to a number larger then 0 (run /usr/local/cpanel/startup afterwards to pickup the changes).

After checking the bandwidth usage for user xxxx, I found nothing wrong with this domain, even if I suspend this domain, server load didn't come down. Please advise what go wrong with the server. I have also change the root password recently to make sure it is secured.

[abubin]

run "top" command and see what services is taking up all the load...

[Stanleytiew]

Thank for your reply, at this moment CPU load is low after I restart the server. Next time when this happen again I will run "top" command and see what taking up CPU load. once I know which service taken up the load, how can I stop the service?

[ngchandak]

Hi,

Once you know the service which make the server load high then just kill that service.

[Stanleytiew]

Thank for your help, Can you tell me what is the command to kill the service.

[ngchandak]

Hi,
First the check which service is making the server load high then kill by the following command.

killall service name

OR

killall -9 service name


Here the service name ie; exim,spamd or any service which make server load.

[kris1351]

Well, it sounds like you have a problem. I just read your 8k+ mails in the mail queue thread and you might just have someone that is running background stuff on your box. I would suggest having someone like Chirpy's company take a look at your server.

[SuperBaby]

kris1351 is right. You are most probably under attack. Exactly the same thing happened to me before.

After checking the bandwidth usage for user xxxx, I found nothing wrong with this domain, even if I suspend this domain, server load didn't come down.

That is misleading. Don't blame the xxxx user. It might not due to his account.

Login as ROOT and check your /tmp folder. Locate any suspicious file. Check if you or any user use any file upload script at the websites. If you do, make sure you limit the file extension allowed and do not allow users to change the chmod values.

A few clue how you can probe further:

Check for any suspicious operation.
root@server01 [~]# ps aux

The process that stands out here is (just an example here):
nobody 17313 0.0 0.0 1452 180 ? S Dec08 0:00 ./s

root@server01 [~]# lsof -p 17313
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
s 17313 nobody cwd DIR 3,3 4096 2 /
s 17313 nobody rtd DIR 3,3 4096 2 /
s 17313 nobody txt REG 3,3 19402 12963616 /home/userxxxxx/public_html/webtools/imageupload/images/s
s 17313 nobody mem REG 3,3 106400 99206 /lib/ld-2.3.2.so
s 17313 nobody mem REG 3,3 1539996 99259 /lib/tls/libc-2.3.2.so
s 17313 nobody 0u CHR 1,3 15 /dev/null
s 17313 nobody 1u CHR 1,3 15 /dev/null
.....

We can see here that the program is located in /home/userxxxxx/public_html/webtools/imageupload/images/s and that the program is listening on TCP port 4000 (You will see this in the network usage section as well). This process is a backdoor!!!

root@server01 [~]# netstat -plntu
tcp 0 0 0.0.0.0:4000 0.0.0.0:* LISTEN 17313/s
tcp 0 0 0.0.0.0:1 0.0.0.0:* LISTEN 4895/portsentry
tcp 0 0 0.0.0.0:993 0.0.0.0:* LISTEN 4757/stunnel-4.04lo
tcp 0 0 0.0.0.0:2082 0.0.0.0:* LISTEN 4791/cpsrvd - waiti

The only suspicious entry it the backdoor listening on port 4000.

[Stanleytiew]

I detect the problem that cause my server load to go maxed but wonder how to solve it permanently. The following is the step taken to solve the problem temporary whenever the server load go maxed:

1. login to WHM and under service manager uncheck eximstats and syslogd
(note: without eximstats and syslogd disable, server load go maxed immediately after
server restart and unable to ssh to server)
2. restart the server
3. ssh to the server and remove file in /usr/local/apache/domlogs
3. go to service manager and enable eximstats and syslogd again

This service has to be done daily otherwise the server load will go up to 18, I just worry server will crash.

In my tweak settings, under Stats and Logs, I have checked "Delete each domain's access logs after stats run"

My biggest problem now is I don't know what go wrong as the domlogs is not clear after stats run, or I also wonder maybe some other service that cause the server to go maxed. I also not sure whether backup cause the problem. Currently I run daily backup to /home/backup.

my server intel Xeon 2.8Ghz with 1GB Ram, /home space use is only 20%. Anybody can please advise me on how to solve this problem.

[kris1351]

The running of domlogs won't cause your server load to go up like that. There is something else wrong and you should hire and admin who can fix it for you.