SOLVED [CPANEL-20407] chkservd restarts MySQL when server time doesn't match MySQL time

[Bigwebmaster]

I have our server configured so that MySQL actually returns UTC time, and this is intentional.

However the server itself has its local time configured as PDT.

It seems like an update recently happened where ChkServd is now checking that the server time and MySQL time both use the same time zone, or it thinks there are issues.

# /scripts/restartsrv_mysql --check                             
(XID 3szs86) The server time and the MySQL® time are different. at /usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm line 156.

I believe this is causing the MySQL server to keep rebooting every time it checks, which seems to be about once every ten minutes.

I would rather keep MySQL set to UTC time which I do in my.cnf:

default-time-zone = '+00:00'

If I look in /usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm:

    if ( $hasmysql && $dbpassword ) {
        no warnings 'once';
        local $SIG{"ALRM"} = sub {
            die Cpanel::Exception::create( 'ConnectionFailed', \%exception_parameters )->to_string();    ## no extract maketext (variable is metadata; the default message will be used)
        };
        alarm(30);
        my $dbh = eval { Cpanel::MysqlUtils::Connect::get_dbi_handle( 'extra_args' => { 'PrintError' => 0, 'RaiseError' => 0 } ) };
        if ( !$dbh ) {
            die Cpanel::Exception::create( 'Services::CheckFailed', [ service => $self->{'service'}, message => $DBI::errstr ] );
        }
        alarm(0);
        die Cpanel::Exception::create( 'ConnectionFailed', \%exception_parameters )->to_string() unless $dbh;    ## no extract maketext (variable is metadata; the default message will be used)

        ## check difference between system time and mysql server time
        my @lt       = localtime( time() );
        my $time_str = sprintf( "%04d-%02d-%02d %02d:%02d:%02d", $lt[5] + 1900, $lt[4] + 1, @lt[ 3, 2, 1, 0 ] );
        my $sql      = qq(SELECT HOUR(TIMEDIFF('$time_str', NOW())) as hour_diff);

        my $hour_diff_detected = eval {
            alarm(30);
            my $sth  = $dbh->prepare($sql);
            my $rv   = $sth->execute();
            my $href = $sth->fetchrow_hashref();
            alarm(0);
            $sth->finish();
            $dbh->disconnect();
            $mysqlok = 1;
            $href->{hour_diff} > 0 ? 1 : 0;
        };

        alarm(0);
        if ($hour_diff_detected) {
            die Cpanel::Exception::create( 'Database::ServerTime', \%exception_parameters )->to_string();
        }
    }

You can see how it throws the exception there.

Thoughts? I may have to disable the chkservd monitor on MySQL because of the forced reboots when there are really no issues from what I can tell other than the time zone discrepancy.

If I comment out this code:

#        if ($hour_diff_detected) { 
#            die Cpanel::Exception::create( 'Database::ServerTime', \%exception_parameters )->to_string(); 
#        }

Then I get this when running the check:

# /scripts/restartsrv_mysql --check                               
The 'mysql' service passed the check: mysql (/usr/sbin/mysqld) is running as mysql with PID 21990 (systemd+/proc check method).

 

[cPanelMichael]

Hello @Bigwebmaster,

Could you verify which version of cPanel & WHM is installed on this system? There's a case in version 70 that may address the issue you have reported:

Fixed case CPANEL-18468: Gracefully handle /etc/localtime not being a symlink.

Thank you.

 

[Bigwebmaster]

We are using: v70.0.34

 

[Bigwebmaster]

I think a better way to compare MySQL time in this script:

/usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm

is to convert everything into UTC first, and then compare that hours match. A user should be able to set MySQL to use by default a different time zone than the server time. Translated both times are actual equal, and your script should only care that they are using truly the same times.

Cpanel updated to version 70.0.34 this morning (on our server), and that is when all these problems started. I am guessing in versions before this the script that checks times either didn't check the hour before, or something changed there in your logic.

 

[cPAusaf]

Hello,

Sorry to hear about the issues being seen. Would it be possible for you to open a support ticket so that we may take a closer look?

How to Open a Technical Support Ticket - cPanel Knowledge Base - cPanel Documentation

Thanks!

 

[Bigwebmaster]

Thanks, created a ticket too to see if that can help resolve the problem.

 

[cPanelMichael]

Hello,

To update, internal case CPANEL-20407 is open to address an issue in cPanel & WHM version 70 where Chkservd will incorrectly detect MySQL as a down service and subsequently restart it in cases where the default-time-zone option in the /etc/my.cnf file is set to a value that doesn't match the server's time. I'll monitor this case and update this thread with more information as it becomes available. In the meantime, the temporary workarounds are to disable monitoring for MySQL using WHM >> Service Manager or to remove the default-time-zone option in the /etc/my.cnf file.

Thank you.

 

[cPanelMichael]

Hello,

To update, this is solved in cPanel & WHM version 70.0.41:

Fixed case CPANEL-20407: Don't restart MySQL if timezone differs from server time.

This version is currently published to the CURRENT build tier, and should reach the RELEASE build tier soon.

Thank you.

 

[cPanelMichael]

Hello,

To update, this is solved in cPanel & WHM version 70.0.41:

Fixed case CPANEL-20407: Don't restart MySQL if timezone differs from server time.

This version is currently published to the CURRENT build tier, and should reach the RELEASE build tier soon.

Thank you.

cPanel & WHM version 70.0.41 is now published to the RELEASE build tier.