The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[Case 161033] MySQL Server Crashing overnight during upcp

Discussion in 'Database Discussions' started by dandanfireman, Feb 5, 2015.

  1. dandanfireman

    dandanfireman Well-Known Member
    PartnerNOC

    Joined:
    May 31, 2002
    Messages:
    117
    Likes Received:
    0
    Trophy Points:
    16
    We had a very eventful night last night. We saw approximately 10-20% of our servers failing mysql checks last night. Each one was found to have just completed running upcp --cron.

    Example log messages below from updatelogs
    Code:
    [20150204.232422]   MySQL55-client-5.5.40-1.cp1148
    [20150204.232502]   MySQL55-test-5.5.40-1.cp1148
    [20150204.232520]   Giving mysqld 10 seconds to exit nicely
    [20150204.232530]   MySQL55-server-5.5.40-1.cp1148
    [20150204.232534]   Starting MySQL using init.d
    [20150204.232538]   Starting MySQL....[  OK  ]
    [20150204.232538]   Giving mysqld 5 seconds to start
    [20150204.232545]   Waiting for “mysql” to restart ……The system failed to open the 'mysql' service's PID file '/var/lib/mysql/server1.serverdomain.com.pid': No such file or directory at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 595.
    [20150204.232551]   Looking for 'mysql' as: /usr/bin/mysql
    [20150204.232551]   Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
    [20150204.232551]   The “mysql” service is down.
    [20150204.232551]   Unable to reset root MySQL password. at /usr/local/cpanel/scripts/securemysql line 151.
    [20150204.232552]   The “mysql” service is down.
     
  2. 24x7server

    24x7server Well-Known Member

    Joined:
    Apr 17, 2013
    Messages:
    1,146
    Likes Received:
    34
    Trophy Points:
    48
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Hello,

    Can you please check MySQL server logs and let us know what exact error logs are getting in that file.
     
  3. durangod

    durangod Well-Known Member

    Joined:
    May 12, 2012
    Messages:
    251
    Likes Received:
    10
    Trophy Points:
    18
    cPanel Access Level:
    Website Owner
    [Case 161033] WHM update knocked my mysql down

    I was about to go to bed but decided to check my email first, all of a sudden i get screen after screen after screen of several hundred cron notices like this


    so then i went into the account cpanel to see what was up and when i tried to get into the db section it get
    so i went into the root whm and i see the new modsecurity question and answered it, restarted mysql and all was well again.

    So it appears that update knocked my mysql offline... just wanted to let you know in case you need to do something
     
  4. ES - George

    ES - George Well-Known Member
    PartnerNOC

    Joined:
    Jun 12, 2011
    Messages:
    142
    Likes Received:
    1
    Trophy Points:
    16
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Indeed - I've seen a number of systems effected by this problem. I assume cPanel developers are aware of the problem by now, and working on a patch :)
     
  5. essell

    essell Registered

    Joined:
    Feb 5, 2015
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    re: [Case 161033] WHM update knocked my mysql down

    Yep. I can report this happened on two separate VPSs that I admin today. Looks like the update script just neglected to restart the MySQL process once it finished. Guess automatic package updates are getting turned off for now until CPanel can respond to the issue.
     
  6. wexler

    wexler Registered

    Joined:
    Feb 5, 2015
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    re: [Case 161033] WHM update knocked my mysql down

    Waking up this morning, I encountered the same issue. MySQL was down, which impacted all websites on the server. MySQL processes showed no processes working. The changelog for cPanel/WHM (Change Log for 11.48.0.8) reflects a recent update occurred earlier this morning and I believe that MySQL was not restarted.
     
  7. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,448
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Re: [Case 161033] WHM update knocked my mysql down

    Thanks for the reports, there have also been several tickets opened about this as I understand it. I've updated this thread with a proper case number to reference via the changelog. A fix is on the way soon.
     
  8. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,448
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Re: [Case 161033] WHM update knocked my mysql down

    Sleep? Oh come on, who's got time for sleep? :p
     
  9. radar81

    radar81 Member

    Joined:
    Feb 5, 2015
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    MySQL Offline

    Hi,

    I have a VPS CENTOS 6.6, WHM 11.48.0 (build 8)

    The MySQL server went offline for an unknown reason, which of course caused the websites to go down. I logged into WHM, restarted the MySQL server and then everything was fine. I want to know what has caused this to make sure it doesn't happen again. Please help.

    Thanks!
     
  10. ciao70

    ciao70 Member

    Joined:
    Nov 3, 2006
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
  11. mlew2

    mlew2 Registered

    Joined:
    Mar 10, 2014
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Re: [Case 161033] WHM update knocked my mysql down

    Now he did say going to bed not to sleep........
     
  12. durangod

    durangod Well-Known Member

    Joined:
    May 12, 2012
    Messages:
    251
    Likes Received:
    10
    Trophy Points:
    18
    cPanel Access Level:
    Website Owner
    Re: [Case 161033] WHM update knocked my mysql down

    i was going to do another thread but since i guess this has to do with the update ill post it here. what are you guys doing to me, everything was running so perfectly for months and now its all crashing down.... sites are down again and i go to add my ip to the whitelist thinking i got blocked for some reason and i cant even add my ip to the cphulk white list it says this

    what am i suppose to do now?

    i disabled cphulk and reenabled, same thing. so now im doing a total system reboot. Word for the devs and management, i personally dont care about eye candy in WHM, i dont care about fancy js buttons or sliders, i care that it works. So dont make it pretty, make it work... thanks

    UPDATE: full reboot got the sites back up... so thats ok but i do need to get that table installed or maybe ill wait to see if there is a patch for this for all of us...
     
    #12 durangod, Feb 5, 2015
    Last edited: Feb 5, 2015
  13. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,448
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
  14. durangod

    durangod Well-Known Member

    Joined:
    May 12, 2012
    Messages:
    251
    Likes Received:
    10
    Trophy Points:
    18
    cPanel Access Level:
    Website Owner
    Re: [Case 161033] WHM update knocked my mysql down

    cool fixed..
     
  15. dalem

    dalem Well-Known Member
    PartnerNOC

    Joined:
    Oct 24, 2003
    Messages:
    2,577
    Likes Received:
    40
    Trophy Points:
    48
    Location:
    SLC
    cPanel Access Level:
    DataCenter Provider
    Re: [Case 161033] WHM update knocked my mysql down

    You can sleep when your dead :)

    I could not resist
     
  16. fullfatdesigns

    Joined:
    Aug 1, 2014
    Messages:
    22
    Likes Received:
    1
    Trophy Points:
    3
    cPanel Access Level:
    Root Administrator
    Re: [Case 161033] WHM update knocked my mysql down

    Hi. Woke up to the same issue of websites not able to connect to mySQL's. A Graceful reboot has sorted it, but will this issue return during the day/weekend or is it fixed?
     
  17. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,448
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
  18. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    651
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello :)

    I've merged these threads.

    cPanel version 11.48.0.9 has been released to address the MySQL restart issue that occurs during the upcp process:

    Fixed case 161033: Make MySQL use an init script to start and stop.

    Thank you.
     
Loading...

Share This Page