The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

server load go to maximum

Discussion in 'General Discussion' started by Stanleytiew, Jul 21, 2005.

  1. Stanleytiew

    Stanleytiew Well-Known Member

    Joined:
    May 9, 2005
    Messages:
    143
    Likes Received:
    0
    Trophy Points:
    16
    I receive an email from root as follow:

    IMPORTANT: Do not ignore this email.
    This is cPanel stats runner on ns1.xxxxxxx.com!
    While processing the log files for user xxxx, the cpu has been maxed out for more then a 6 hour period. The current load/uptime line on the server at the time of this email is
    07:41:52 up 11:28, 0 users, load average: 5.00, 5.00, 5.00
    You should check the server to see why the load is so high and take steps to lower the load. If you want stats to continue to run even with a high load; Edit /var/cpanel/cpanel.config and change extracpus to a number larger then 0 (run /usr/local/cpanel/startup afterwards to pickup the changes).

    After checking the bandwidth usage for user xxxx, I found nothing wrong with this domain, even if I suspend this domain, server load didn't come down. Please advise what go wrong with the server. I have also change the root password recently to make sure it is secured.
     
  2. abubin

    abubin Well-Known Member

    Joined:
    Dec 7, 2004
    Messages:
    393
    Likes Received:
    1
    Trophy Points:
    18
    run "top" command and see what services is taking up all the load...
     
  3. Stanleytiew

    Stanleytiew Well-Known Member

    Joined:
    May 9, 2005
    Messages:
    143
    Likes Received:
    0
    Trophy Points:
    16
    Thank for your reply, at this moment CPU load is low after I restart the server. Next time when this happen again I will run "top" command and see what taking up CPU load. once I know which service taken up the load, how can I stop the service?
     
  4. ngchandak

    ngchandak Well-Known Member

    Joined:
    Apr 5, 2005
    Messages:
    54
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    India
    Hi,

    Once you know the service which make the server load high then just kill that service.
     
  5. Stanleytiew

    Stanleytiew Well-Known Member

    Joined:
    May 9, 2005
    Messages:
    143
    Likes Received:
    0
    Trophy Points:
    16
    Thank for your help, Can you tell me what is the command to kill the service.
     
  6. ngchandak

    ngchandak Well-Known Member

    Joined:
    Apr 5, 2005
    Messages:
    54
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    India
    Hi,
    First the check which service is making the server load high then kill by the following command.

    killall service name

    OR

    killall -9 service name


    Here the service name ie; exim,spamd or any service which make server load.
     
  7. kris1351

    kris1351 Well-Known Member

    Joined:
    Apr 18, 2003
    Messages:
    963
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Lewisville, Tx
    Well, it sounds like you have a problem. I just read your 8k+ mails in the mail queue thread and you might just have someone that is running background stuff on your box. I would suggest having someone like Chirpy's company take a look at your server.
     
  8. SuperBaby

    SuperBaby Well-Known Member

    Joined:
    Nov 27, 2003
    Messages:
    331
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Thailand
    cPanel Access Level:
    Website Owner
    Twitter:
    kris1351 is right. You are most probably under attack. Exactly the same thing happened to me before.

    That is misleading. Don't blame the xxxx user. It might not due to his account.

    Login as ROOT and check your /tmp folder. Locate any suspicious file. Check if you or any user use any file upload script at the websites. If you do, make sure you limit the file extension allowed and do not allow users to change the chmod values.

    A few clue how you can probe further:

    Check for any suspicious operation.
    root@server01 [~]# ps aux

    The process that stands out here is (just an example here):
    nobody 17313 0.0 0.0 1452 180 ? S Dec08 0:00 ./s

    root@server01 [~]# lsof -p 17313
    COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
    s 17313 nobody cwd DIR 3,3 4096 2 /
    s 17313 nobody rtd DIR 3,3 4096 2 /
    s 17313 nobody txt REG 3,3 19402 12963616 /home/userxxxxx/public_html/webtools/imageupload/images/s
    s 17313 nobody mem REG 3,3 106400 99206 /lib/ld-2.3.2.so
    s 17313 nobody mem REG 3,3 1539996 99259 /lib/tls/libc-2.3.2.so
    s 17313 nobody 0u CHR 1,3 15 /dev/null
    s 17313 nobody 1u CHR 1,3 15 /dev/null
    .....

    We can see here that the program is located in /home/userxxxxx/public_html/webtools/imageupload/images/s and that the program is listening on TCP port 4000 (You will see this in the network usage section as well). This process is a backdoor!!!

    root@server01 [~]# netstat -plntu
    tcp 0 0 0.0.0.0:4000 0.0.0.0:* LISTEN 17313/s
    tcp 0 0 0.0.0.0:1 0.0.0.0:* LISTEN 4895/portsentry
    tcp 0 0 0.0.0.0:993 0.0.0.0:* LISTEN 4757/stunnel-4.04lo
    tcp 0 0 0.0.0.0:2082 0.0.0.0:* LISTEN 4791/cpsrvd - waiti

    The only suspicious entry it the backdoor listening on port 4000.
     
  9. Stanleytiew

    Stanleytiew Well-Known Member

    Joined:
    May 9, 2005
    Messages:
    143
    Likes Received:
    0
    Trophy Points:
    16
    I detect the problem that cause my server load to go maxed but wonder how to solve it permanently. The following is the step taken to solve the problem temporary whenever the server load go maxed:

    1. login to WHM and under service manager uncheck eximstats and syslogd
    (note: without eximstats and syslogd disable, server load go maxed immediately after
    server restart and unable to ssh to server)
    2. restart the server
    3. ssh to the server and remove file in /usr/local/apache/domlogs
    3. go to service manager and enable eximstats and syslogd again

    This service has to be done daily otherwise the server load will go up to 18, I just worry server will crash.

    In my tweak settings, under Stats and Logs, I have checked "Delete each domain's access logs after stats run"

    My biggest problem now is I don't know what go wrong as the domlogs is not clear after stats run, or I also wonder maybe some other service that cause the server to go maxed. I also not sure whether backup cause the problem. Currently I run daily backup to /home/backup.

    my server intel Xeon 2.8Ghz with 1GB Ram, /home space use is only 20%. Anybody can please advise me on how to solve this problem.
     
    #9 Stanleytiew, Jul 27, 2005
    Last edited: Jul 27, 2005
  10. kris1351

    kris1351 Well-Known Member

    Joined:
    Apr 18, 2003
    Messages:
    963
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Lewisville, Tx
    The running of domlogs won't cause your server load to go up like that. There is something else wrong and you should hire and admin who can fix it for you.
     
Loading...

Share This Page