The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

cpanel mysql update breaks mysql, in yet another way

Discussion in 'Database Discussions' started by cornernote, Apr 30, 2014.

  1. cornernote

    cornernote Member

    Joined:
    Nov 9, 2003
    Messages:
    24
    Likes Received:
    0
    Trophy Points:
    6
    cpanel wouldnt update at all, cos mysql was too old, so i allowed it to update once... big mistake

    old my.cnf:
    binlog-do-db=db1
    binlog-do-db=db2
    binlog-do-db=db3
    binlog-do-db=db4

    new my.cnf:
    binlog-do-db=db4

    ouch...

    guess now i'll do a full mysql backup, copy it to all the slaves, and start the reload process.
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,762
    Likes Received:
    662
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello :)

    Could you elaborate on the specific method you used to upgrade MySQL? Was it through Web Host Manager? What version did you upgrade to and from?

    Thank you.
     
  3. cornernote

    cornernote Member

    Joined:
    Nov 9, 2003
    Messages:
    24
    Likes Received:
    0
    Trophy Points:
    6
    Hi cPanelMichael,

    I don't know where to find that information. The details of the update may be listed in your ticket #4810867
     
  4. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,762
    Likes Received:
    662
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    That ticket addresses the cPanel update itself. Could you open a separate ticket regarding the changes made to your /etc/my.cnf file so we can take a closer look? You can post the ticket number here so we can update this thread with the outcome.

    Thank you.
     
  5. cornernote

    cornernote Member

    Joined:
    Nov 9, 2003
    Messages:
    24
    Likes Received:
    0
    Trophy Points:
    6
    Hi cPanelMichael,

    Really, I'd prefer if cpanel staff didn't access the server. It's clear to me that the cpanel upgrade tried adding to the my.cnf file, by taking all the settings and removing the duplicates. cPanel seems completely unaware that some settings are supposed to be duplicated.

    cPanel is great if you don't know how to configure mysql for yourself, however if you run a server with advanced settings then you have to be very careful that it does not break your mysql.

    The damage is already done, and now I am moving our servers away from cpanel to prevent any future damage.
     
  6. vanessa

    vanessa Well-Known Member
    PartnerNOC

    Joined:
    Sep 26, 2006
    Messages:
    817
    Likes Received:
    22
    Trophy Points:
    18
    Location:
    Virginia Beach, VA
    cPanel Access Level:
    DataCenter Provider

    I've really never seen this happen - and I run hundreds of servers with custom my.cnf settings. You should let cPanel troubleshoot the problem if you're going to complain about it on their forums.
     
  7. cornernote

    cornernote Member

    Joined:
    Nov 9, 2003
    Messages:
    24
    Likes Received:
    0
    Trophy Points:
    6
    @vanessa,

    Do you have binlog-do-db multiple times (or any other similar setting such as replicate-do-db)?

    If you want some (but not all) databases in your binlog, you have to define binlog-do-db on multiple lines.

    Do you use binary logging or replication at all? If you do, its amazing you have had no issues. It has broken replication during updates before, see this blog post for the exact details:
    mysql_install_db, mysqld –bootstrap, binary log, cPanel | Open Query blog
    http://bugs.mysql.com/bug.php?id=43398

    If you don't use binary logging or replication (eg, using cpanel for toy websites), it probably works fine. If you use it for real websites that need advanced features such as binary logging and replication then look out, it will cause you no end of grief.

    Edit: Also, your conclusion does not follow from your premise. Why should I grant access to my server in order to be able to "complain" on this forum? I have provided details of the issue. If they are unable to understand the problem then I sure as heck am not letting them touch my box. They should have the resources to setup a server of their own, set the my.cnf as I have described, then run the mysql update. It seems they have not attempted to do that.
     
    #7 cornernote, May 2, 2014
    Last edited: May 2, 2014
  8. vanessa

    vanessa Well-Known Member
    PartnerNOC

    Joined:
    Sep 26, 2006
    Messages:
    817
    Likes Received:
    22
    Trophy Points:
    18
    Location:
    Virginia Beach, VA
    cPanel Access Level:
    DataCenter Provider
    I actually have four servers that I manage where replication is used in this fashion, but not necessarily by choice :)

    Quick reminder: avoid using binlog-do-db | code.openark.org
    Why MySQL's binlog-do-db option is dangerous - MySQL Performance Blog

    I haven't seen a problem so far where these lines were removed following a MySQL update. I suppose if it's that big of an issue, chattr'ing the my.cnf isn't completely out of the question, but typically any admin would advise against doing that unless absolutely necessary. You can alternatively prevent cPanel from updating MySQL at all, and instead run the updates on your own terms when you are able to monitor what is going on. I'm not saying that this isn't a bug or that cPanel isn't doing this - I'm simply saying that *I* have not seen this problem while using a similar setup.

    Regarding my latter statement, you may have misunderstood me. If you're going to make a stink about a problem and even open a ticket for them to look into it, you should be willing to give them the ability to actually help. Sure, cPanel reps have the ability to create test environments, and they do. The few times I do contact cPanel support, I've experienced nothing less than them going above and beyond to resolve my issues, even when I contact them through "unofficial" channels. However, often times when troubleshooting they will need to be able to evaluate the environment in which the problem is occurring. Not every server is set up identically, and you've already indicated that you have a fairly advanced setup that their test environment is unlikely going to mimic out of the box. So if you're not going to let them troubleshoot the problem, you can't really accuse them of being incompetent or not understanding.
     
  9. cornernote

    cornernote Member

    Joined:
    Nov 9, 2003
    Messages:
    24
    Likes Received:
    0
    Trophy Points:
    6
    @vanessa,

    I am well aware of the issues with using binlog-do-db, and my application has been developed with these things in mind. Replication has never been broken by the issues you pointed out, nor do they have anything to do with this bug report. Replication has however been broken by the cpanel mysql updater on multiple occasions.

    As specified in my OP I do disable the mysql auto-updater because of previous problems which I linked to, and you seem to have no comment on. The author of the article described cpanel's work arounds as dangerous hacks that seemed to lack understanding of the cause to the actual problem.

    Replicating the issue should be very simple. I have explained the specific parts of the my.cnf that get removed. If they run a test and it doesn't happen for them, they are welcome to report that in this post. This has not been done, so I assume this has not been tested. I don't see how access to my box will assist them with that. After all they can't re-upgrade mysql. Even if they could, I don't want them to. If they need more information about the box I am be happy to provide it, however this has not been requested.

    I don't want to get further this debate with you about how I should configure my server, who should have access, or if cpanel did or didn't cause the issue. The facts are simple, cpanel's updater did not consider how the my.cnf can be configured, and broke mysql replication. I have every right to post this bug report here in the forum, and every right to choose who has access to my box. I wish you would stop using terms like "complain" and "make a stink". You seem to be trying to belittle my bug report. It's upto cpanel if they wish to address the issue or not. So please go about your day instead of trying to hijack this thread.
     
  10. vanessa

    vanessa Well-Known Member
    PartnerNOC

    Joined:
    Sep 26, 2006
    Messages:
    817
    Likes Received:
    22
    Trophy Points:
    18
    Location:
    Virginia Beach, VA
    cPanel Access Level:
    DataCenter Provider
    yo - don't post to a public forum if you don't want sensible replies. If you say something inaccurate or ignorant, expect someone to correct you. You also didn't post any of what you stated in your OP, in fact, the OP was barely readable. Chill.
     
  11. cornernote

    cornernote Member

    Joined:
    Nov 9, 2003
    Messages:
    24
    Likes Received:
    0
    Trophy Points:
    6
    I have no issue with sensible replies, and feel free to correct me if I am wrong about something (however not sure what needs correcting). So far you have only told me that I should give access to my server (with no good reason), I shouldn't use binlog-do-db (again, no good reason), stated that you don't have this problem (without clarification if you use binlog-do-db), and used terms to make it seem like I am making a fuss, when in fact I am simply reporting this as an issue.

    OP says "so i allowed it to update once". OP explains the issue that binlog-do-db was wiped from my.cnf.

    Are you trying to help with this thread, or just trying to increase your post count? Perhaps you can explain what is is you are trying to help with. Your comments are not assisting to get this issue resolved.

    In fact they are doing the opposite because you are advocating that cpanel staff should not investigate unless I give them server access. I can't see why they need it, nor has anyone given me any reason why they need it.
     
  12. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,465
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Defects should be reported via this link:
    http://go.cpanel.net/bugs

    That link is on the top of every page of this forum.

    Now that your cPanel was updated for you by cPanel Technical Support, and assuming you've reset your custom settings, have you tried forcing another cPanel update just to see if you can duplicate the issue of the my.cnf being modified again? What was the result?

    If cPanel update is "completely unaware" of some setting as you suggest, then, you opening a Defect report ticket and letting cPanel Technical Support back in to verify this situation and resolve it, is important. Not just for you, but to better the product itself.

    This solves nothing.


    What troubles me is this:


    With this setup that you require, you updated without knowing what you were updating to? How can that be?

    I am trying to help, and my post count is fine. vanessa and cPanelMichael are trying to help you too.

    cPanel Technical Support is standing by, waiting to help you as well.
    http://go.cpanel.net/bugs


    Good luck with this!
     
  13. cornernote

    cornernote Member

    Joined:
    Nov 9, 2003
    Messages:
    24
    Likes Received:
    0
    Trophy Points:
    6
    Hi @Infopro,

    Thanks for pointing me to the defects page. I didn't see it. I'll report the issue there.

    I didnt try updating again, because I don't want to risk breaking replication again. I am running a live site that requires real-time replication to be functional.

    I prefer cpanel staff to not come onto my server because it increases the risk that something will be changed from zero to something above zero. If you need to verify something then you can try on your own non-live servers, or if you need further information you can ask me and I will gladly provide it.

    I prefer for cpanel staff not to resolve anything, because I have already rectified the problem that was caused. I am very concerned for the state of my servers and don't need additional changes at this point. If it is simply to better the cpanel product then testing should be performed away from my live environment. Suggesting testing on my live server increases the chance that something will go wrong, with no advantage other than to test and improve a product that I have already stated I won't be using any more.

    I am taking the time to report this issue, but I am not putting my server on the line, and quite frankly I am getting fed up with the persistent requests for access and lack of information requested, that I offered to provide if only a specific request was made. Can you please clarify what it is you need to check on my server and I will gladly check. The only thing that has been asked so far is the previous and new version numbers. I stated that I don't know where to find these, and no instructions have been given to obtain this information. If the techs login, where will they get this information?

    Personally I think moving away from cpanel will prevent future issues. Can you clarify why you think it solves nothing?

    I didn't think it was required to record the previous and upgraded version of cpanel. Apparently the tech that assisted with the upgrade felt the same way. In addition it seems cpanel itself doesn't store that information, so it seems that its not considered important until after the fact. If its recorded in a log file then you can point me to it and I'll update this thread with the information.

    I have expressed my concern with venessa's replies. I have no concern about cPanelMichael's post and didn't claim he was not trying to help. Not sure why you added his name or your name to the list after quoting my comment, seems you are concerned I will find those posts non-helpful, but that is not the case.
     
  14. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,465
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    When you open a Defect report, you will be asked by cPanel Technical Support for access. You could of course provide specific details as well so that they can do some testing of their own, but the access needed is so they can view your existing setup.


    The specific question asked was:

    Moving away from cPanel prevents you and your users from using cPanel. Of course if you have no need for cPanel that's something else all together I guess. Finding out what went wrong and solving the issues you had going forward, I think, a better plan.

    The version is at the top of the log file you pasted to your prior ticket. You were on an outdated, EOL version of cPanel, 11.38.

    He asked you a question trying to help, and you choose not to answer that, instead you stated you did not want to provide access, and then ended your reply with:

    You manually killed the update. cPanel sorted this and got you going again. cPanel Technical Support is here to help.

    I'm very happy to hear that you got updated and away from 11.38 and that you sorted out your issues seemingly brought on by the upgrade. Still, I can only assume that you're not the only one with the setup you have, and if somethings not working properly with the cPanel product, cPanel Technical Support should know about it. As I said, to make the product better. You moving away from cPanel doesn't help.

    Whether you provide access or not is your call of course, but please do open a new ticket on this, as Michael suggested, and provide as many details as you can so that it could be duplicated, tested and any issues that are found, can get fixed.

    I have read no comments by cPanel Technical Support here or in your prior ticket that mention if your existing setup is even supported as you have it. If it's not, I would think taking precautions ahead of updating might have been prudent.

    GL!
     
    vanessa likes this.
  15. cPanelPeter

    cPanelPeter Technical Analyst III
    Staff Member

    Joined:
    Sep 23, 2013
    Messages:
    569
    Likes Received:
    15
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    So I spun up a test server running 11.38.2 (build 25), and MySQL 5.0.96

    Code:
    -bash-4.1# /usr/local/cpanel/cpanel -V
    11.38.2 (build 25)
    -bash-4.1# mysql -V
    mysql  Ver 14.12 Distrib 5.0.96, for unknown-linux-gnu (x86_64) using readline 5.1
    -bash-4.1#
    
    I then modified the /etc/my.cnf file and added 4 lines:

    Code:
    # cat my.cnf 
    [mysqld]
    innodb_file_per_table=1
    
    binlog-do-db=db1
    binlog-do-db=db2
    binlog-do-db=db3
    binlog-do-db=db4
    
    -bash-4.1#
    
    I'm assuming here that I don't actually have to have any replication running, since you simply stated that the my.cnf file is changed during a MySQL update. But if that's wrong, then please do let me know.

    Then I ran a MySQL Update. Which updated MySQL to 5.1.73

    Code:
    -bash-4.1# mysql -V
    mysql  Ver 14.14 Distrib 5.1.73, for unknown-linux-gnu (x86_64) using readline 5.1
    -bash-4.1# 
    
    I then checked the /etc/my.cnf file and it still looked identical to before the update...

    Code:
    -bash-4.1# cat /etc/my.cnf
    [mysqld]
    innodb_file_per_table=1
    
    binlog-do-db=db1
    binlog-do-db=db2
    binlog-do-db=db3
    binlog-do-db=db4
    
    -bash-4.1# 
    
    So I was not able to duplicate the problem you originally reported.

    I will try again now updating to MySQL 5.5, and will post the results shortly.
     
  16. cPanelPeter

    cPanelPeter Technical Analyst III
    Staff Member

    Joined:
    Sep 23, 2013
    Messages:
    569
    Likes Received:
    15
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    I was able to reproduce the issue you experienced when updating from 5.1 to 5.5.

    The /etc/my.cnf file now does have those additional options missing...

    Code:
    -bash-4.1# cat /etc/my.cnf
    [mysqld]
    innodb_file_per_table=1
    binlog-do-db=db4
    default-storage-engine=MyISAM
    -bash-4.1# 
    
    Now that I know I can reproduce the issue, I will file an internal case on this with our developers.
     
  17. vanessa

    vanessa Well-Known Member
    PartnerNOC

    Joined:
    Sep 26, 2006
    Messages:
    817
    Likes Received:
    22
    Trophy Points:
    18
    Location:
    Virginia Beach, VA
    cPanel Access Level:
    DataCenter Provider
    Perhaps having this info from the OP would have helped narrow it down - it would explain why it's unfamiliar, because not only is this a non-standard setup, but he's going from a very outdated version.

    I would assume then that this isn't an issue anymore for the OP, since upgrading from 5.1 to 5.5 is typically a one-time thing.
     
  18. cPanelPeter

    cPanelPeter Technical Analyst III
    Staff Member

    Joined:
    Sep 23, 2013
    Messages:
    569
    Likes Received:
    15
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    While that's true (MySQL updates should be pretty rare) and many of these back and forth conversations could have been avoided with all the information at hand, the problem still needs to be addressed. I just checked again and this happens with an update to MySQL 5.6 as well (running 11.42.1.5).

    So I have opened an internal case on this issue. Case # 100717 - MySQL Update from 5.1 to 5.5/5.6 removes lines in /etc/my.cnf that can then break MySQL

    When our developers address this issue, it will appear in the change logs.

    I do want to thank everyone for their input and helping me get this bug tested and confirmed. Although it may not seem like it, all the responses were indeed helpful.
     
  19. vanessa

    vanessa Well-Known Member
    PartnerNOC

    Joined:
    Sep 26, 2006
    Messages:
    817
    Likes Received:
    22
    Trophy Points:
    18
    Location:
    Virginia Beach, VA
    cPanel Access Level:
    DataCenter Provider
    Thanks @cPanelPeter :)
     
  20. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,465
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    What she said. :p
     
Loading...

Share This Page