The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Services (Apache, MySQL, etc.) crashing on server

Discussion in 'General Discussion' started by Ioan Sameli, Feb 22, 2015.

  1. Ioan Sameli

    Ioan Sameli Member

    Joined:
    Jan 20, 2015
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Shibuya, Tokyo, Japan
    cPanel Access Level:
    Root Administrator
    All our websites went down over the week-end. The server was responding (could login in SSH) but all the services were down. At some point during the week-end I got a client report a database connection error on their site (meaning Apache was running but MySQL crashed at some point - this morning they were all down).

    A reboot fixed everything, but it's the second time it happens in a few weeks.

    I checked the Daily Process Log and no process took more than 5% of mem or cpu over the last few days. No website had any traffic peak. The whole setup runs on a decent EC2 instance (m3.medium with 3.75GB of ram), I doubt it's a hardware fault. It's a standard LAMP installation and we haven't added any 3rd party software.

    • What could be the reason those service crashes? Are there specific logs I could analyze?
    • Is there a simple way to monitor of MySQL / Apache and get an alert when they crash?
    • Any other advice on how to prevent this to happen, or at least understand what it comes from?

    Thanks in advance for any help!
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello :)

    You can ensure monitoring for your services are enabled via:

    "WHM Home » Service Configuration » Service Manager"

    Check the Apache and MySQL error logs and review the time at which those services crashed to see if you can find any useful information:

    Apache error log - /usr/local/apache/logs/error_log
    MySQL error log - /var/lib/mysql/$hostname.err

    Thank you.
     
  3. Ioan Sameli

    Ioan Sameli Member

    Joined:
    Jan 20, 2015
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Shibuya, Tokyo, Japan
    cPanel Access Level:
    Root Administrator
    Hi Michael,

    I found something very rotten in the MySQL error log:

    150220 6:12:41 [Warning] IP address '122.226.102.45' could not be resolved: Name or service not known
    150220 17:57:16 [Warning] IP address '117.21.176.129' could not be resolved: Name or service not known
    150221 0:16:29 [Warning] IP address '61.240.144.66' could not be resolved: Temporary failure in name resolution
    150221 10:56:32 [Warning] IP address '118.122.252.71' could not be resolved: Name or service not known
    150224 13:46:10 [Warning] IP address '61.240.144.64' could not be resolved: Temporary failure in name resolution


    The IPs are from China, and the time correspond to when the server crashed. Not sure why it made the server crash though.

    I've realised my MySQL server was open to the world (I'm surprised this is the default cPanel config, it seems really insecure). I limited connections to localhost by adding bind-address = 127.0.0.1 in the my.cnf.

    I'm not sure how this can crash MySQL though? Would brute force attack crash the MySQL server? Could it also lead to crashing the other services such as Apache and BIND?
     
  4. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Note that MySQL is not open by default to the public. The remote IP address must be added as an access host via:

    "WHM Home » SQL Services » Additional MySQL Access Hosts"

    Do you see those same entries every time it fails, or was that a one time occurrence?

    Thank you.
     
  5. Ioan Sameli

    Ioan Sameli Member

    Joined:
    Jan 20, 2015
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Shibuya, Tokyo, Japan
    cPanel Access Level:
    Root Administrator
    Right. Then do you have any idea about how those entries end up in the log? How can the Chinese IP do connection attempts on the SQL server if it doesn't accept external connections?

    The server crashed twice so far, and twice there were those in the log at approximatively the time it crashed. Not sure if they're the cause, but they seem related.

    Thanks a lot for your support!
     
  6. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    The connections are allowed, but authorization will fail. You can disable inbound connections to port 3306 in your firewall if you want to block these connections completely, which is likely a good idea if those IP addresses are attempting to brute force your databases.

    Thank you.
     
Loading...

Share This Page