The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

MySQL down - chkserv.d never catch this.

Discussion in 'General Discussion' started by ispro, Oct 28, 2004.

  1. ispro

    ispro Well-Known Member

    Joined:
    Apr 8, 2004
    Messages:
    628
    Likes Received:
    1
    Trophy Points:
    18
    I have a problem.
    chkservd.d doesn't catch if mysql is really responding.

    For example, I'm removing /var/lib/mysql/HOSTNAME.pid while mysql not stopped.
    Then starting mysql (another copy!) service mysql start
    Then check it:
    mysql
    ERROR 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
    It's nice.

    But chkserv.d not catch that MySQL is already down.
    service chkservd restart
    doesn't help.

    I've tuned /etc/chkserv.d/mysql to the following (to remove old mysql process):
    killall -TERM mysql;sleep 2;killall -9 -g mysql;killall -TERM mysqld;sleep 2;killall -9 -g mysqld;/scripts/restartsrv mysql

    However looks like a problem in how chkserv.d threat its checks. Not in command to execute.

    May anyone help.
     
  2. Sinewy

    Sinewy Well-Known Member

    Joined:
    May 15, 2004
    Messages:
    367
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Sydney, Australia
    cPanel Access Level:
    DataCenter Provider
    as a rule of thumb: removing the mysql pid file kills the socket. not recommended.
     
  3. ispro

    ispro Well-Known Member

    Joined:
    Apr 8, 2004
    Messages:
    628
    Likes Received:
    1
    Trophy Points:
    18
    Damn! I knew this!
    It was the modelled situation which chkserv.d cannot to handle.

    I'm asking about how to make chkserv.d to deal with such a situations!
     
  4. netlook

    netlook Well-Known Member
    PartnerNOC

    Joined:
    Mar 25, 2004
    Messages:
    335
    Likes Received:
    0
    Trophy Points:
    16
    Yeah, good question. Can anybody say something about this? How to solve the problem? I also have this situation - there are opened too much mysqld procesess on my server and then mysql stops working with information "Too many connections" - but chkservd didn't catch this... Any idea?
     
    #4 netlook, Oct 28, 2004
    Last edited: Oct 29, 2004
  5. MattDr2

    MattDr2 Well-Known Member
    PartnerNOC

    Joined:
    Feb 19, 2003
    Messages:
    84
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Norman, OK
    We are also looking for a resolve. This has become a big problem lately.

    Regards,
    Matt
     
  6. ispro

    ispro Well-Known Member

    Joined:
    Apr 8, 2004
    Messages:
    628
    Likes Received:
    1
    Trophy Points:
    18
    I would like to share the simple piece of code that helps me to at least restart MySQL (and chkservd additionally). Remember to have /root/.my.cnf with MySQL access info! (cPanel servers has it already).

    Put in the cron under root user and you will be happy :)

    Code:
    #!/usr/bin/perl
    
    print "Checking if the common services up & running\n";
    mysql();
    chkservd();
    
    # MySQL
    sub mysql {
    print "\n*** Checking if the MySQLrunning... ***\n\n";
    $mysql_status = `mysqladmin status`;
    print $mysql_status;
    if ($mysql_status =~ m/Uptime/)
        {
        print "\nMySQL uptime ok.\n";
        }
    else
        {
        print "\nMySQL down/unreachable! Going to hard-reboot it...\n";
        `killall -TERM mysql;sleep 2;killall -9 -g mysql;killall -TERM mysqld;sleep 2;killall -9 -g mysqld;/scripts/restartsrv mysql`;
        }
    }
    
    # chkservd
    sub chkservd {
    print "\n*** Checking if the chkservd running... ***\n\n";
    $chkservd_status = `service chkservd status`;
    print $chkservd_status;
    if ($chkservd_status =~ m/running/)
        {
        print "\nchkservd running.\n";
        }
    else
        {
        print "\nchkservd stopped! Going to restart it...\n";
        `service chkservd restart`;
        }
    }
    
    
     
  7. raventec

    raventec Well-Known Member

    Joined:
    Apr 19, 2003
    Messages:
    120
    Likes Received:
    0
    Trophy Points:
    16
    Neat!

    :) Neat! Thanks for that!
     
  8. netlook

    netlook Well-Known Member
    PartnerNOC

    Joined:
    Mar 25, 2004
    Messages:
    335
    Likes Received:
    0
    Trophy Points:
    16
    Great!

    :) :) :)

    Very Very Very Thank You!
     
  9. cPanelNick

    cPanelNick Administrator
    Staff Member

    Joined:
    Mar 9, 2015
    Messages:
    3,426
    Likes Received:
    2
    Trophy Points:
    38
    cPanel Access Level:
    DataCenter Provider
    We are going to switch back to the method of checking the mysql server status instead of checking if its running or not in the next release of /scripts.
     
  10. cPanelNick

    cPanelNick Administrator
    Staff Member

    Joined:
    Mar 9, 2015
    Messages:
    3,426
    Likes Received:
    2
    Trophy Points:
    38
    cPanel Access Level:
    DataCenter Provider
    You can get this update by running /scripts/updatenow
     
  11. ispro

    ispro Well-Known Member

    Joined:
    Apr 8, 2004
    Messages:
    628
    Likes Received:
    1
    Trophy Points:
    18
    What exactly updated by this?
    chkserv.d or something else?

    Btw, I've noticed that chkserv.d constantly restarts MySQL (I have removed all the other monitoring scripts already!). I was insisted to stop chkservd service!

    What can be the case?..
     
    #11 ispro, Oct 29, 2004
    Last edited: Oct 29, 2004
  12. netlook

    netlook Well-Known Member
    PartnerNOC

    Joined:
    Mar 25, 2004
    Messages:
    335
    Likes Received:
    0
    Trophy Points:
    16
    How is it called and how it works? Or how to make it working?
     
  13. cPanelNick

    cPanelNick Administrator
    Staff Member

    Joined:
    Mar 9, 2015
    Messages:
    3,426
    Likes Received:
    2
    Trophy Points:
    38
    cPanel Access Level:
    DataCenter Provider
    check /var/log/chkservd.log
     
  14. manghooli

    manghooli Well-Known Member

    Joined:
    Aug 22, 2004
    Messages:
    55
    Likes Received:
    0
    Trophy Points:
    6
    released?!
     
  15. BenThomas

    BenThomas Well-Known Member

    Joined:
    Feb 12, 2004
    Messages:
    598
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Houston, Texas USA
    cPanel Access Level:
    Root Administrator
    Yes, it is released. Check /scripts/restartsrv_mysql for the actual code. Checksrvd calls /scripts/restartsrv_mysql --check to determine is mysql is running. Before it was just checking for mysql processes, now it makes a connection to the server.
     
  16. netlook

    netlook Well-Known Member
    PartnerNOC

    Joined:
    Mar 25, 2004
    Messages:
    335
    Likes Received:
    0
    Trophy Points:
    16
    Ok, everything is working fine when I manually run this script, but when I put it into cron with:

    */5 * * * * root /path/to/script

    I recive an e-mail notification with:

    mysqladmin: connect to server at 'localhost' failed
    error: 'Access denied for user: 'root@localhost' (Using password: NO)'


    How to prevent this?
     
  17. ispro

    ispro Well-Known Member

    Joined:
    Apr 8, 2004
    Messages:
    628
    Likes Received:
    1
    Trophy Points:
    18
    Add "HOME=/root" before cron job

    e.g.:
    HOME=/root
    */5 * * * * root /pathto/script >/dev/null 2>&1

    Then it will use /root/.my.cnf to connect to MySQL
     
  18. netlook

    netlook Well-Known Member
    PartnerNOC

    Joined:
    Mar 25, 2004
    Messages:
    335
    Likes Received:
    0
    Trophy Points:
    16
    Of course :p I forgot... Shame on me :eek:

    But another question - also after running script from cron with fixed issue .my.cnf, in confirmation mail there is wrote that chkservd is restarted, because it wasn't running. Its not true. When I run the script manually form SSH everything is ok. Any idea?

    Thank you
     
  19. ispro

    ispro Well-Known Member

    Joined:
    Apr 8, 2004
    Messages:
    628
    Likes Received:
    1
    Trophy Points:
    18
    I have no idea on that...
    Add the $chkservd_status to the mail body to see if it has got a reply of "service chkservd status".

    We have no problems with that. Even if chkservd failed, our script restart it smoothly.
     
  20. netlook

    netlook Well-Known Member
    PartnerNOC

    Joined:
    Mar 25, 2004
    Messages:
    335
    Likes Received:
    0
    Trophy Points:
    16
    Ok, thank you :)
     
Loading...

Share This Page