The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

VPS Down What logs file need to be checked

Discussion in 'General Discussion' started by lldeepakll, Jul 22, 2012.

  1. lldeepakll

    lldeepakll Well-Known Member

    Joined:
    May 20, 2012
    Messages:
    86
    Likes Received:
    2
    Trophy Points:
    8
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Hi,

    I have a VPS that was suddenly down, and now it is working fine. I want to find out the reason what causes the vps to down. So please tell me what logs files I should check to find out the reason of server down or how to check what process was consuming high cpu resources ?
     
  2. CitizenK

    CitizenK Well-Known Member

    Joined:
    Jun 5, 2012
    Messages:
    64
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    On The Road
    cPanel Access Level:
    Root Administrator
    Hello,

    Below are some of the common logfiles you will want to check when looking for the cause of errors. If you have questions about any of these logs please let us know, and if possible include the part of the log file in question.

    Code:
    System Messages: /var/log/messages
    MySQL Error Log: /var/lib/mysql/hostname.err  (replace hostname with your servers hostname)
    Server Status Log: /var/log/chkservd.log
    cPanel Main Error Log: /usr/local/cpanel/logs/error_log
    Maillog: /var/log/maillog
    Apache: /usr/local/apache/domlogs/
    
     
  3. lldeepakll

    lldeepakll Well-Known Member

    Joined:
    May 20, 2012
    Messages:
    86
    Likes Received:
    2
    Trophy Points:
    8
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Thanks for your reply. I could find the below logs that seems to me causes the problem, can you please suggest.
    In /var/log/chkservd.log
    Code:
    [2012-07-21 04:11:06 -0500] Disk check .... /dev/sda3 (/) [7%] ... /dev/sdb1 (/disk2) [18%] ... /dev/sda1 (/boot) [82%] ... {status:ok} ... Done
    [2012-07-21 04:11:06 -0500] Service check ....syslogd [[check command:+][socket connect:N/A]]...sshd [[check command:+][socket connect:N/A]]...spamd [[check command:+][socket connect:N/A]]...queueprocd [[check command:+][socket connect:N/A]]...named [[check command:+][socket connect:N/A]]...mysql [[check command:+][socket connect:N/A]]...mailman [[check command:+][socket connect:N/A]]...lfd [[check command:+][socket connect:N/A]]...ipaliases [[check command:+][socket connect:N/A]]...imap [[socket_service_auth:1][check command:+][socket connect:+]]...httpd [Timeout while trying to get data from service: Died at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 607.
    [2012-07-21 05:51:45 -0500] Disk check .... /dev/sda3 (/) [7%] ... /dev/sdb1 (/disk2) [18%] ... /dev/sda1 (/boot) [82%] ... {status:ok} ... Done
    [2012-07-21 05:51:45 -0500] Service check ....syslogd [[check command:+][socket connect:N/A]]...sshd [[check command:+][socket connect:N/A]]...spamd [too soon after restart to check]...queueprocd [[check command:+][socket connect:N/A]]...named [[check command:+][socket connect:N/A]]...mysql [[check command:+][socket connect:N/A]]...mailman [[check command:-][socket connect:N/A][fail count:1]Restarting mailman....
    [2012-07-21 06:32:43 -0500] Disk check .... /dev/sda3 (/) [7%] ... /dev/sdb1 (/disk2) [18%] ... /dev/sda1 (/boot) [82%] ... {status:ok} ... Done
    [2012-07-21 06:32:43 -0500] Service check ....syslogd [[check command:+][socket connect:N/A]]...sshd [[check command:+][socket connect:N/A]]...spamd [too soon after restart to check]...queueprocd [[check command:+][socket connect:N/A]]...named [[check command:+][socket connect:N/A]]...mysql [[check command:+][socket connect:N/A]]...mailman [[check command:-][socket connect:N/A][fail count:1]Restarting mailman....
    [2012-07-21 06:40:49 -0500] Disk check .... /dev/sda3 (/) [7%] ... /dev/sdb1 (/disk2) [18%] ... /dev/sda1 (/boot) [82%] ... {status:ok} ... Done
    [2012-07-21 06:40:49 -0500] Service check ....syslogd [[check command:+][socket connect:N/A]]...sshd [[check command:+][socket connect:N/A]]...spamd [Service Check Interrupted
    [2012-07-21 07:02:15 -0500] Disk check .... /dev/sda3 (/) [7%] ... /dev/sdb1 (/disk2) [18%] ... /dev/sda1 (/boot) [82%] ... {status:ok} ... Done
    [2012-07-21 07:02:15 -0500] Service check ....syslogd [[check command:+][socket connect:N/A]]...sshd [[check command:+][socket connect:N/A]]...spamd [too soon after restart to check]...queueprocd [[check command:+][socket connect:N/A]]...named [[check command:+][socket connect:N/A]]...mysql [[check command:+][socket connect:N/A]]...mailman [[check command:-][socket connect:N/A][fail count:1]Restarting mailman....
    [2012-07-21 07:08:02 -0500] Disk check .... /dev/sda3 (/) [7%] ... /dev/sdb1 (/disk2) [18%] ... /dev/sda1 (/boot) [82%] ... {status:ok} ... Done
    
    In /var/log/messages
    Code:
    Jul 21 04:06:21 ehostrus pure-ftpd: (jaipur@115.119.174.18) [NOTICE] /home/jaipur//www/advt/Hotel-Indiana-Classic-200X200.html uploaded  (421 bytes, 1.48KB/sec)
    Jul 21 04:06:21 ehostrus pure-ftpd: (jaipur@115.119.174.18) [INFO] Can't change directory to /www/advt/Hotel-Indiana-Classic-200X200.jpg: No such file or directory
    Jul 21 04:06:24 ehostrus pure-ftpd: (jaipur@115.119.174.18) [NOTICE] /home/jaipur//www/advt/Hotel-Indiana-Classic-200X200.jpg uploaded  (33034 bytes, 27.42KB/sec)
    Jul 21 04:06:56 ehostrus pure-ftpd: (jaipur@115.119.174.18) [INFO] Logout.
    Jul 21 04:11:44 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=1 ID=32373 PROTO=UDP SPT=42660 DPT=33470 LEN=40 
    Jul 21 04:11:44 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=2 ID=32374 PROTO=UDP SPT=44423 DPT=33471 LEN=40 
    Jul 21 04:11:44 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=1 ID=32375 PROTO=UDP SPT=51997 DPT=33472 LEN=40 
    Jul 21 04:11:44 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=2 ID=32376 PROTO=UDP SPT=43773 DPT=33473 LEN=40 
    Jul 21 04:11:44 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=3 ID=32377 PROTO=UDP SPT=44876 DPT=33474 LEN=40 
    Jul 21 04:11:50 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=5 ID=32386 PROTO=UDP SPT=36698 DPT=33483 LEN=40 
    Jul 21 04:11:50 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=7 ID=32388 PROTO=UDP SPT=59209 DPT=33485 LEN=40 
    Jul 21 04:11:55 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=11 ID=32402 PROTO=UDP SPT=58468 DPT=33499 LEN=40 
    Jul 21 04:11:55 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=11 ID=32404 PROTO=UDP SPT=37256 DPT=33501 LEN=40 
    Jul 21 04:11:55 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=12 ID=32405 PROTO=UDP SPT=44281 DPT=33502 LEN=40 
    Jul 21 04:12:00 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=16 ID=32419 PROTO=UDP SPT=45804 DPT=33516 LEN=40 
    Jul 21 04:12:00 ehostrus kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=188.138.124.110 DST=184.154.204.123 LEN=60 TOS=0x00 PREC=0x00 TTL=16 ID=32418 PROTO=UDP SPT=48229 DPT=33515 LEN=40 
    Jul 21 04:13:22 ehostrus pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Jul 21 04:13:58 ehostrus pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__O8CnRne2jxzfDnnZ9r53jiDg4EoCQw3wK54nFd1SRs2jrww18LvAzIeMsmp6fPHV is now logged in
    Jul 21 04:13:59 ehostrus pure-ftpd: (__cpanel__service__auth__ftpd__O8CnRne2jxzfDnnZ9r53jiDg4EoCQw3wK54nFd1SRs2jrww18LvAzIeMsmp6fPHV@127.0.0.1) [INFO] Logout.
    Jul 21 04:15:47 ehostrus kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
    Jul 21 04:15:48 ehostrus kernel: 
    Jul 21 04:15:48 ehostrus kernel: Call Trace:
    Jul 21 04:15:48 ehostrus kernel:  [<ffffffff800d4a52>] out_of_memory+0x9f/0x25c
    Jul 21 04:15:48 ehostrus kernel:  [<ffffffff8000f15f>] __alloc_pages+0x2a0/0x34d
    Jul 21 04:15:48 ehostrus kernel:  [<ffffffff80154b24>] __first_cpu+0xe/0x1d
    Jul 21 04:15:48 ehostrus kernel:  [<ffffffff80012c5b>] __do_page_cache_readahead+0x96/0x18e
    Jul 21 04:15:48 ehostrus kernel:  [<ffffffff8001364d>] filemap_nopage+0x14c/0x386
    Jul 21 04:15:48 ehostrus kernel:  [<ffffffff800082e4>] __handle_mm_fault+0x26f/0xd9b
    Jul 21 04:15:48 ehostrus kernel:  [<ffffffff8006555d>] thread_return+0x6a/0x177
    Jul 21 04:15:48 ehostrus kernel:  [<ffffffff800693d8>] do_page_fault+0x4c9/0x846
    Jul 21 04:15:48 ehostrus kernel:  [<ffffffff8005edff>] del_timer_sync+0xc/0x16
    Jul 21 04:16:02 ehostrus kernel:  [<ffffffff8009a6a9>] process_timeout+0x0/0x5
    Jul 21 04:16:03 ehostrus pure-ftpd: (jaipur@115.119.174.18) [INFO] Logout.
    Jul 21 04:16:45 ehostrus kernel:  [<ffffffff8010af98>] sys_epoll_wait+0x3b8/0x3f9
    Jul 21 04:18:37 ehostrus kernel:  [<ffffffff80061e39>] error_exit+0x0/0x84
    Jul 21 04:19:44 ehostrus kernel: 
    Jul 21 04:20:34 ehostrus kernel: Mem-info:
    Jul 21 04:20:48 ehostrus kernel: Node 0 DMA per-cpu:
    Jul 21 04:21:45 ehostrus kernel: cpu 0 hot: high 0, batch 1 used:0
    Jul 21 04:22:43 ehostrus kernel: cpu 0 cold: high 0, batch 1 used:0
    Jul 21 04:23:00 ehostrus kernel: cpu 1 hot: high 0, batch 1 used:0
    Jul 21 04:23:16 ehostrus kernel: cpu 1 cold: high 0, batch 1 used:0
    Jul 21 04:23:34 ehostrus kernel: cpu 2 hot: high 0, batch 1 used:0
    Jul 21 04:23:41 ehostrus kernel: cpu 2 cold: high 0, batch 1 used:0
    Jul 21 04:24:09 ehostrus kernel: cpu 3 hot: high 0, batch 1 used:0
    Jul 21 07:04:00 ehostrus kernel: Firewall: *ICMP_IN Blocked* IN=eth0 OUT= MAC=00:25:90:74:fc:2a:00:15:63:3f:f9:e4:08:00 SRC=74.217.89.116 DST=184.154.204.123 LEN=84 TOS=0x00 PREC=0x00 TTL=48 ID=58521 PROTO=ICMP TYPE=8 CODE=0 ID=13830 SEQ=9291 
    Jul 21 07:04:02 ehostrus kernel: Firewall: *ICMP_IN Blocked* IN=eth0 OUT= MAC=00:25:90
     
  4. CitizenK

    CitizenK Well-Known Member

    Joined:
    Jun 5, 2012
    Messages:
    64
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    On The Road
    cPanel Access Level:
    Root Administrator
    Hello,

    I did not see any issues in the logs you posted, however I recommend that if you want a cPanel Tech to take a deeper look at your system that you open a ticket at https://tickets.cpanel.net
     
  5. lldeepakll

    lldeepakll Well-Known Member

    Joined:
    May 20, 2012
    Messages:
    86
    Likes Received:
    2
    Trophy Points:
    8
    Location:
    India
    cPanel Access Level:
    Root Administrator
    OK. Can you please guide me about this process. (
    PHP:
     <defunct>). I have noticed that this process taking too much cpu memory
    USER     PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME   COMMAND

    ehostr    432915  16.0  0.0      0     0 
    ?        ZN   06:42       0:00         [php] <defunct>
     
  6. CitizenK

    CitizenK Well-Known Member

    Joined:
    Jun 5, 2012
    Messages:
    64
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    On The Road
    cPanel Access Level:
    Root Administrator
    if you notice that php processes are running too long and consuming too many resources this is commonly caused my a excessive value for "max_execution_time"

    In WHM >> Service Configuration >> PHP Configuration Editor and check to see if the "max_execution_time" is set above 30 and the max_input_time is set above 60. If they are above these values you may want to consider lowering them. ß
     
  7. lldeepakll

    lldeepakll Well-Known Member

    Joined:
    May 20, 2012
    Messages:
    86
    Likes Received:
    2
    Trophy Points:
    8
    Location:
    India
    cPanel Access Level:
    Root Administrator
    I will check the same. Thanks for your support cpCitizenK :)
     
  8. CitizenK

    CitizenK Well-Known Member

    Joined:
    Jun 5, 2012
    Messages:
    64
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    On The Road
    cPanel Access Level:
    Root Administrator
    Thanks! Let us know if that helped.
     
  9. lldeepakll

    lldeepakll Well-Known Member

    Joined:
    May 20, 2012
    Messages:
    86
    Likes Received:
    2
    Trophy Points:
    8
    Location:
    India
    cPanel Access Level:
    Root Administrator
    I have checked and max_execution_time is already set 30 to and max_input_time is set to 60. Do I need to increase these value ? Further I have killed that process.
     
  10. CitizenK

    CitizenK Well-Known Member

    Joined:
    Jun 5, 2012
    Messages:
    64
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    On The Road
    cPanel Access Level:
    Root Administrator
    No, I would not recommend increasing these values. Unless you have a specific need, it is best to keep these at the default values I posted above.
     
  11. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    The issue where you see "httpd invoked oom-killer" means you've run out of memory on the system at that point in time. I discuss that error specifically:

    http://forums.cpanel.net/f5/out-memory-283892.html#post1186502

    It's most definitely an issue to run out of memory. That log entry in /var/log/messages is definitely important on what happened.
     
  12. lldeepakll

    lldeepakll Well-Known Member

    Joined:
    May 20, 2012
    Messages:
    86
    Likes Received:
    2
    Trophy Points:
    8
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Hi Tristan,

    I did not find the keyword "incorrect key" in /var/lib/mysql/*.err as you suggested. But I have found the below logs in /var/log/messages just before the "Jul 21 04:15:47 ehostrus kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
    Jul 21 04:15:48 ehostrus kernel:
    Jul 21 04:15:48 ehostrus kernel: Call Trace:
    Jul 21 04:15:48 ehostrus kernel: [<ffffffff800d4a52>] out_of_memory+0x9f/0x25c"

    Can you please assist about below logs.

    Jul 21 03:56:46 ehostrus suhosin[862623]: ALERT - configured GET variable value length limit exceeded - dropped variable 'maneref' (attacker '203.88.22.141', file

    '/home/japur/public_html/index.php')
    Jul 21 03:56:46 ehostrus suhosin[862623]: ALERT - configured GET variable value length limit exceeded - dropped variable 'maneref' (attacker '203.88.22.141', file

    '/home/japur/public_html/index.php')
    Jul 21 03:56:51 ehostrus suhosin[862628]: ALERT - configured GET variable value length limit exceeded - dropped variable 'maneref' (attacker '203.88.22.141', file

    '/home/japur/public_html/index.php')
    Jul 21 03:56:51 ehostrus last message repeated 2 times
     
  13. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    The logs simply show suhosin dropping a connection due to the GET variable being too long for the IP 203.88.22.141 to page /home/japur/public_html/index.php

    I'm not certain unless there were a lot of these at the time that it could have caused the issue for memory depletion. You might have a DoS type attack going on if that's the case. CSF, ddos-deflate and mod_qos can all be used to help in the case of a denial of service attack.
     
Loading...

Share This Page