The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

My Servers down for 20 minutes

Discussion in 'General Discussion' started by sniperscope, Jul 19, 2011.

  1. sniperscope

    sniperscope Well-Known Member

    Joined:
    Apr 5, 2011
    Messages:
    118
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    osaka/ japan
    cPanel Access Level:
    Website Owner
    Hello
    Last night i figured out that all my servers went down about 20 minutes. After that became normal all of sudden.
    When i checked whm today i found some graphs in munin reports.
    Also last night there was big storm in japan might it cause it?

    Here is some Screen shots

    Apache
    [​IMG]

    CPU
    [​IMG]

    eth0
    [​IMG]

    MySQL
    [​IMG]

    Processes
    [​IMG]

    VMstat
    [​IMG]

    How can i find what did go wrong? Or what caused this?
     
    #1 sniperscope, Jul 19, 2011
    Last edited: Jul 19, 2011
  2. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Check "last" on the machine to see if the server actually went down. If it didn't go down, then it wasn't that the server wasn't running but something else like network connectivity or latency.

    You could review /var/log/messages around the timeframe in question to see what might have happened.
     
  3. sniperscope

    sniperscope Well-Known Member

    Joined:
    Apr 5, 2011
    Messages:
    118
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    osaka/ japan
    cPanel Access Level:
    Website Owner
    Dear Tristan
    Here i have /var/log/messages output is: You don't have permission

    # cd /var
    [/var]# cd /log and output is: There is no such a file or directory
     
  4. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    You cannot use / when cding to a directory within a directory:

    Code:
    cd /var
    cd log
    That would work, but what you tried would not work. I'm uncertain why you aren't just doing:

    Code:
    cd /var/log
    grep time messages
    Where time is the time when the server was down. Otherwise, you could open the file:

    Code:
    vi /var/log/messages
    Then search for the time once opened.
     
  5. sniperscope

    sniperscope Well-Known Member

    Joined:
    Apr 5, 2011
    Messages:
    118
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    osaka/ japan
    cPanel Access Level:
    Website Owner
    Jul 20 00:00:02 server1 suhosin[28276]: ALERT - script tried to increase memory_limit to 536870912 bytes which is above the allowed value (attacker 'REMOTE_ADDR not set', file '/home/winhosti/public_html/billing/admin/cron.php', line 0)
    Jul 20 00:02:20 server1 init: Id "x" respawning too fast: disabled for 5 minutes
    Jul 20 00:03:02 server1 suhosin[31818]: ALERT - configured request variable name length limit exceeded - dropped variable '/home/winhosti/public_html/billing/modules/registrars/resellerclub/resellerclubsync_php' (attacker 'REMOTE_ADDR not set', file 'unknown')
    Jul 20 00:07:21 server1 init: Id "x" respawning too fast: disabled for 5 minutes
    Jul 20 00:12:23 server1 init: Id "x" respawning too fast: disabled for 5 minutes

    Here it is the result.
    Is WHMCS cause this? Because billing sub domain is belong to whmcs.

    I want to ask you one more thing because this messages reminds me something.
    At first time i was using WHMCS and set cron jobs etc... but now i am using my own built system how can i disable/delete whmcs cron jobs?
     
  6. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    If whmcs has cron jobs set anywhere in /var/spool/cron location, you can remove them in the file(s) there. I don't know where WHMCS places the cron jobs. If they are not in that location, then you might want to post at WHMCS Forums - Powered by vBulletin location or search there for further information.

    As for the results, did "last" show the server didn't reboot then? If so, then what did you see besides the Munin results provided? Was the server hanging at the time(s) in question or were pages not displaying for multiple sites? I just wondered, since the /var/log/messages output only indicates an issue with one specific account (winhosti).

    You might want to also check /usr/local/apache/logs/error_log during the same timeframe.
     
  7. sniperscope

    sniperscope Well-Known Member

    Joined:
    Apr 5, 2011
    Messages:
    118
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    osaka/ japan
    cPanel Access Level:
    Website Owner
    not only winhosti, all of accounts and all servers were down. Server1, server2 and server3 i could not even ping to 8.8.8.8

    Command you provided gave me thousands of result and seems all of them typical html error
     
  8. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    What command did I provide? I said to check the log for the exact timeframe when the server was down in the log at /usr/local/apache/logs/error_log location. If there were 1000s of hits in a few minutes timeframe, that was a pretty active machine.
     
  9. sniperscope

    sniperscope Well-Known Member

    Joined:
    Apr 5, 2011
    Messages:
    118
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    osaka/ japan
    cPanel Access Level:
    Website Owner
    sorry, i got what you mean.
    here is the result :


    [Wed Jul 20 00:00:07 2011] [error] [client 219.67.4.57] client denied by server configuration: /home/uhclub/public_html/crossdomain.xml
    [Wed Jul 20 00:00:07 2011] [error] [client 219.67.4.57] File does not exist: /home/uhclub/public_html/403.shtml
    [Wed Jul 20 00:00:09 2011] [error] [client 219.67.4.57] client denied by server configuration: /home/uhclub/public_html/crossdomain.xml
    [Wed Jul 20 00:00:09 2011] [error] [client 219.67.4.57] File does not exist: /home/uhclub/public_html/403.shtml
    [Wed Jul 20 00:00:15 2011] [error] [client 61.125.122.87] File does not exist: /home/osakaesp/public_html/access/images, referer:
    [Wed Jul 20 00:00:15 2011] [error] [client 61.125.122.87] File does not exist: /home/osakaesp/public_html/404.shtml, referer:
    [Wed Jul 20 00:00:16 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/system/images, referer:
    [Wed Jul 20 00:00:16 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/404.shtml, referer: *
    [Wed Jul 20 00:00:20 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/system/images, referer:
    [Wed Jul 20 00:00:20 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/404.shtml, referer:
    [Wed Jul 20 00:00:34 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/common/img/icon_0.gif, referer:
    [Wed Jul 20 00:00:34 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/404.shtml, referer:
    [Wed Jul 20 00:00:57 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/common/img/icon_0.gif, referer:
    [Wed Jul 20 00:00:57 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/404.shtml, referer:
    [Wed Jul 20 00:01:10 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/common/img/icon_0.gif, referer:
    [Wed Jul 20 00:01:10 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/404.shtml, referer:
    [Wed Jul 20 00:01:21 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/system/images, referer:
    [Wed Jul 20 00:01:21 2011] [error] [client 180.18.46.237] File does not exist: /home/osakaesp/public_html/404.shtml, referer: *
    [Wed Jul 20 00:16:04 2011] [notice] Graceful restart requested, doing restart
    [Wed Jul 20 00:16:05 2011] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
    [Wed Jul 20 00:16:05 2011] [notice] Apache/2.2.18 (Unix) mod_ssl/2.2.18 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 configured -- resuming normal operations
    [Wed Jul 20 00:17:00 2011] [notice] caught SIGTERM, shutting down
    [Wed Jul 20 00:17:02 2011] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
    [Wed Jul 20 00:17:02 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
    [Wed Jul 20 00:17:02 2011] [notice] ModSecurity for Apache/2.5.13 (ModSecurity: Open Source Web Application Firewall) configured.
    [Wed Jul 20 00:17:03 2011] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
    [Wed Jul 20 00:17:03 2011] [notice] Apache/2.2.18 (Unix) mod_ssl/2.2.18 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 configured -- resuming normal operations


    As you can see somethings start become weird at 00:16:04

    Edit: URL tag was edited
     
    #9 sniperscope, Jul 19, 2011
    Last edited: Jul 19, 2011
Loading...

Share This Page