Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

SOLVED [CPANEL-20407] chkservd restarts MySQL when server time doesn't match MySQL time

Discussion in 'Database Discussion' started by Bigwebmaster, May 11, 2018.

  1. Bigwebmaster

    Bigwebmaster Active Member

    Joined:
    Dec 3, 2003
    Messages:
    31
    Likes Received:
    9
    Trophy Points:
    158
    I have our server configured so that MySQL actually returns UTC time, and this is intentional.

    However the server itself has its local time configured as PDT.

    It seems like an update recently happened where ChkServd is now checking that the server time and MySQL time both use the same time zone, or it thinks there are issues.

    Code:
    # /scripts/restartsrv_mysql --check                             
    (XID 3szs86) The server time and the MySQL® time are different. at /usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm line 156.
    
    I believe this is causing the MySQL server to keep rebooting every time it checks, which seems to be about once every ten minutes.

    I would rather keep MySQL set to UTC time which I do in my.cnf:

    Code:
    default-time-zone = '+00:00'
    If I look in /usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm:

    Code:
        if ( $hasmysql && $dbpassword ) {
            no warnings 'once';
            local $SIG{"ALRM"} = sub {
                die Cpanel::Exception::create( 'ConnectionFailed', \%exception_parameters )->to_string();    ## no extract maketext (variable is metadata; the default message will be used)
            };
            alarm(30);
            my $dbh = eval { Cpanel::MysqlUtils::Connect::get_dbi_handle( 'extra_args' => { 'PrintError' => 0, 'RaiseError' => 0 } ) };
            if ( !$dbh ) {
                die Cpanel::Exception::create( 'Services::CheckFailed', [ service => $self->{'service'}, message => $DBI::errstr ] );
            }
            alarm(0);
            die Cpanel::Exception::create( 'ConnectionFailed', \%exception_parameters )->to_string() unless $dbh;    ## no extract maketext (variable is metadata; the default message will be used)
    
            ## check difference between system time and mysql server time
            my @lt       = localtime( time() );
            my $time_str = sprintf( "%04d-%02d-%02d %02d:%02d:%02d", $lt[5] + 1900, $lt[4] + 1, @lt[ 3, 2, 1, 0 ] );
            my $sql      = qq(SELECT HOUR(TIMEDIFF('$time_str', NOW())) as hour_diff);
    
            my $hour_diff_detected = eval {
                alarm(30);
                my $sth  = $dbh->prepare($sql);
                my $rv   = $sth->execute();
                my $href = $sth->fetchrow_hashref();
                alarm(0);
                $sth->finish();
                $dbh->disconnect();
                $mysqlok = 1;
                $href->{hour_diff} > 0 ? 1 : 0;
            };
    
            alarm(0);
            if ($hour_diff_detected) {
                die Cpanel::Exception::create( 'Database::ServerTime', \%exception_parameters )->to_string();
            }
        }
    
    You can see how it throws the exception there.

    Thoughts? I may have to disable the chkservd monitor on MySQL because of the forced reboots when there are really no issues from what I can tell other than the time zone discrepancy.

    If I comment out this code:

    Code:
    #        if ($hour_diff_detected) { 
    #            die Cpanel::Exception::create( 'Database::ServerTime', \%exception_parameters )->to_string(); 
    #        }
    Then I get this when running the check:

    Code:
    # /scripts/restartsrv_mysql --check                               
    The 'mysql' service passed the check: mysql (/usr/sbin/mysqld) is running as mysql with PID 21990 (systemd+/proc check method).
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #1 Bigwebmaster, May 11, 2018
    Last edited: May 11, 2018
  2. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,196
    Likes Received:
    1,936
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @Bigwebmaster,

    Could you verify which version of cPanel & WHM is installed on this system? There's a case in version 70 that may address the issue you have reported:

    Fixed case CPANEL-18468: Gracefully handle /etc/localtime not being a symlink.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Bigwebmaster

    Bigwebmaster Active Member

    Joined:
    Dec 3, 2003
    Messages:
    31
    Likes Received:
    9
    Trophy Points:
    158
    We are using: v70.0.34
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. Bigwebmaster

    Bigwebmaster Active Member

    Joined:
    Dec 3, 2003
    Messages:
    31
    Likes Received:
    9
    Trophy Points:
    158
    I think a better way to compare MySQL time in this script:

    /usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm

    is to convert everything into UTC first, and then compare that hours match. A user should be able to set MySQL to use by default a different time zone than the server time. Translated both times are actual equal, and your script should only care that they are using truly the same times.

    Cpanel updated to version 70.0.34 this morning (on our server), and that is when all these problems started. I am guessing in versions before this the script that checks times either didn't check the hour before, or something changed there in your logic.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. cPAusaf

    cPAusaf Linux Technical Analyst II
    Staff Member

    Joined:
    Aug 24, 2016
    Messages:
    25
    Likes Received:
    1
    Trophy Points:
    78
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. Bigwebmaster

    Bigwebmaster Active Member

    Joined:
    Dec 3, 2003
    Messages:
    31
    Likes Received:
    9
    Trophy Points:
    158
    Thanks, created a ticket too to see if that can help resolve the problem.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    cPanelMichael likes this.
  7. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,196
    Likes Received:
    1,936
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    To update, internal case CPANEL-20407 is open to address an issue in cPanel & WHM version 70 where Chkservd will incorrectly detect MySQL as a down service and subsequently restart it in cases where the default-time-zone option in the /etc/my.cnf file is set to a value that doesn't match the server's time. I'll monitor this case and update this thread with more information as it becomes available. In the meantime, the temporary workarounds are to disable monitoring for MySQL using WHM >> Service Manager or to remove the default-time-zone option in the /etc/my.cnf file.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,196
    Likes Received:
    1,936
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    To update, this is solved in cPanel & WHM version 70.0.41:

    Fixed case CPANEL-20407: Don't restart MySQL if timezone differs from server time.

    This version is currently published to the CURRENT build tier, and should reach the RELEASE build tier soon.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,196
    Likes Received:
    1,936
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    cPanel & WHM version 70.0.41 is now published to the RELEASE build tier.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice