Services (Apache, MySQL, etc.) crashing on server

Ioan Sameli

Member
Jan 20, 2015
11
0
1
Shibuya, Tokyo, Japan
cPanel Access Level
Root Administrator
All our websites went down over the week-end. The server was responding (could login in SSH) but all the services were down. At some point during the week-end I got a client report a database connection error on their site (meaning Apache was running but MySQL crashed at some point - this morning they were all down).

A reboot fixed everything, but it's the second time it happens in a few weeks.

I checked the Daily Process Log and no process took more than 5% of mem or cpu over the last few days. No website had any traffic peak. The whole setup runs on a decent EC2 instance (m3.medium with 3.75GB of ram), I doubt it's a hardware fault. It's a standard LAMP installation and we haven't added any 3rd party software.

  • What could be the reason those service crashes? Are there specific logs I could analyze?
  • Is there a simple way to monitor of MySQL / Apache and get an alert when they crash?
  • Any other advice on how to prevent this to happen, or at least understand what it comes from?

Thanks in advance for any help!
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello :)

You can ensure monitoring for your services are enabled via:

"WHM Home » Service Configuration » Service Manager"

Check the Apache and MySQL error logs and review the time at which those services crashed to see if you can find any useful information:

Apache error log - /usr/local/apache/logs/error_log
MySQL error log - /var/lib/mysql/$hostname.err

Thank you.
 

Ioan Sameli

Member
Jan 20, 2015
11
0
1
Shibuya, Tokyo, Japan
cPanel Access Level
Root Administrator
Hi Michael,

I found something very rotten in the MySQL error log:

150220 6:12:41 [Warning] IP address '122.226.102.45' could not be resolved: Name or service not known
150220 17:57:16 [Warning] IP address '117.21.176.129' could not be resolved: Name or service not known
150221 0:16:29 [Warning] IP address '61.240.144.66' could not be resolved: Temporary failure in name resolution
150221 10:56:32 [Warning] IP address '118.122.252.71' could not be resolved: Name or service not known
150224 13:46:10 [Warning] IP address '61.240.144.64' could not be resolved: Temporary failure in name resolution


The IPs are from China, and the time correspond to when the server crashed. Not sure why it made the server crash though.

I've realised my MySQL server was open to the world (I'm surprised this is the default cPanel config, it seems really insecure). I limited connections to localhost by adding bind-address = 127.0.0.1 in the my.cnf.

I'm not sure how this can crash MySQL though? Would brute force attack crash the MySQL server? Could it also lead to crashing the other services such as Apache and BIND?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Note that MySQL is not open by default to the public. The remote IP address must be added as an access host via:

"WHM Home » SQL Services » Additional MySQL Access Hosts"

Do you see those same entries every time it fails, or was that a one time occurrence?

Thank you.
 

Ioan Sameli

Member
Jan 20, 2015
11
0
1
Shibuya, Tokyo, Japan
cPanel Access Level
Root Administrator
Note that MySQL is not open by default to the public.
Right. Then do you have any idea about how those entries end up in the log? How can the Chinese IP do connection attempts on the SQL server if it doesn't accept external connections?

The server crashed twice so far, and twice there were those in the log at approximatively the time it crashed. Not sure if they're the cause, but they seem related.

Thanks a lot for your support!
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
The connections are allowed, but authorization will fail. You can disable inbound connections to port 3306 in your firewall if you want to block these connections completely, which is likely a good idea if those IP addresses are attempting to brute force your databases.

Thank you.