[Case 161033] MySQL Server Crashing overnight during upcp

dandanfireman

Well-Known Member
PartnerNOC
May 31, 2002
117
0
316
We had a very eventful night last night. We saw approximately 10-20% of our servers failing mysql checks last night. Each one was found to have just completed running upcp --cron.

Example log messages below from updatelogs
Code:
[20150204.232422]   MySQL55-client-5.5.40-1.cp1148
[20150204.232502]   MySQL55-test-5.5.40-1.cp1148
[20150204.232520]   Giving mysqld 10 seconds to exit nicely
[20150204.232530]   MySQL55-server-5.5.40-1.cp1148
[20150204.232534]   Starting MySQL using init.d
[20150204.232538]   Starting MySQL....[  OK  ]
[20150204.232538]   Giving mysqld 5 seconds to start
[20150204.232545]   Waiting for “mysql” to restart ……The system failed to open the 'mysql' service's PID file '/var/lib/mysql/server1.serverdomain.com.pid': No such file or directory at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 595.
[20150204.232551]   Looking for 'mysql' as: /usr/bin/mysql
[20150204.232551]   Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
[20150204.232551]   The “mysql” service is down.
[20150204.232551]   Unable to reset root MySQL password. at /usr/local/cpanel/scripts/securemysql line 151.
[20150204.232552]   The “mysql” service is down.
 

durangod

Well-Known Member
May 12, 2012
504
46
78
cPanel Access Level
Website Owner
[Case 161033] WHM update knocked my mysql down

I was about to go to bed but decided to check my email first, all of a sudden i get screen after screen after screen of several hundred cron notices like this


<b>Warning</b>: mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in
so then i went into the account cpanel to see what was up and when i tried to get into the db section it get
Error while connecting to MySQL: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) Error while connecting to MySQL: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) at /usr/local/cpanel/Cpanel/Mysql.pm line 164.
so i went into the root whm and i see the new modsecurity question and answered it, restarted mysql and all was well again.

So it appears that update knocked my mysql offline... just wanted to let you know in case you need to do something
 

essell

Registered
Feb 5, 2015
1
0
1
cPanel Access Level
Root Administrator
re: [Case 161033] WHM update knocked my mysql down

Yep. I can report this happened on two separate VPSs that I admin today. Looks like the update script just neglected to restart the MySQL process once it finished. Guess automatic package updates are getting turned off for now until CPanel can respond to the issue.
 

wexler

Registered
Feb 5, 2015
1
0
1
cPanel Access Level
Root Administrator
re: [Case 161033] WHM update knocked my mysql down

Waking up this morning, I encountered the same issue. MySQL was down, which impacted all websites on the server. MySQL processes showed no processes working. The changelog for cPanel/WHM (Change Log for 11.48.0.8) reflects a recent update occurred earlier this morning and I believe that MySQL was not restarted.
 

Infopro

Well-Known Member
May 20, 2003
17,091
516
613
Pennsylvania
cPanel Access Level
Root Administrator
Twitter
Re: [Case 161033] WHM update knocked my mysql down

Thanks for the reports, there have also been several tickets opened about this as I understand it. I've updated this thread with a proper case number to reference via the changelog. A fix is on the way soon.
 

radar81

Member
Feb 5, 2015
7
0
1
cPanel Access Level
Root Administrator
MySQL Offline

Hi,

I have a VPS CENTOS 6.6, WHM 11.48.0 (build 8)

The MySQL server went offline for an unknown reason, which of course caused the websites to go down. I logged into WHM, restarted the MySQL server and then everything was fine. I want to know what has caused this to make sure it doesn't happen again. Please help.

Thanks!
 

durangod

Well-Known Member
May 12, 2012
504
46
78
cPanel Access Level
Website Owner
Re: [Case 161033] WHM update knocked my mysql down

i was going to do another thread but since i guess this has to do with the update ill post it here. what are you guys doing to me, everything was running so perfectly for months and now its all crashing down.... sites are down again and i go to add my ip to the whitelist thinking i got blocked for some reason and i cant even add my ip to the cphulk white list it says this

The system failed to add an IP address to cphulkd hosts because the “cphulkd.ip_lists” table may be corrupted and returned the following error: DBD::mysql::db do failed: Table 'cphulkd.ip_lists' doesn't exist.
what am i suppose to do now?

i disabled cphulk and reenabled, same thing. so now im doing a total system reboot. Word for the devs and management, i personally dont care about eye candy in WHM, i dont care about fancy js buttons or sliders, i care that it works. So dont make it pretty, make it work... thanks

UPDATE: full reboot got the sites back up... so thats ok but i do need to get that table installed or maybe ill wait to see if there is a patch for this for all of us...
 
Last edited:

durangod

Well-Known Member
May 12, 2012
504
46
78
cPanel Access Level
Website Owner
Re: [Case 161033] WHM update knocked my mysql down

cool fixed..
 

fullfatdesigns

Well-Known Member
Aug 1, 2014
72
11
8
cPanel Access Level
Root Administrator
Re: [Case 161033] WHM update knocked my mysql down

Hi. Woke up to the same issue of websites not able to connect to mySQL's. A Graceful reboot has sorted it, but will this issue return during the day/weekend or is it fixed?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,228
463
Hello :)

I've merged these threads.

cPanel version 11.48.0.9 has been released to address the MySQL restart issue that occurs during the upcp process:

Fixed case 161033: Make MySQL use an init script to start and stop.

Thank you.