SOLVED [CPANEL-20407] chkservd restarts MySQL when server time doesn't match MySQL time

Bigwebmaster

Active Member
Dec 3, 2003
32
9
158
I have our server configured so that MySQL actually returns UTC time, and this is intentional.

However the server itself has its local time configured as PDT.

It seems like an update recently happened where ChkServd is now checking that the server time and MySQL time both use the same time zone, or it thinks there are issues.

Code:
# /scripts/restartsrv_mysql --check                             
(XID 3szs86) The server time and the MySQL® time are different. at /usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm line 156.
I believe this is causing the MySQL server to keep rebooting every time it checks, which seems to be about once every ten minutes.

I would rather keep MySQL set to UTC time which I do in my.cnf:

Code:
default-time-zone = '+00:00'
If I look in /usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm:

Code:
    if ( $hasmysql && $dbpassword ) {
        no warnings 'once';
        local $SIG{"ALRM"} = sub {
            die Cpanel::Exception::create( 'ConnectionFailed', \%exception_parameters )->to_string();    ## no extract maketext (variable is metadata; the default message will be used)
        };
        alarm(30);
        my $dbh = eval { Cpanel::MysqlUtils::Connect::get_dbi_handle( 'extra_args' => { 'PrintError' => 0, 'RaiseError' => 0 } ) };
        if ( !$dbh ) {
            die Cpanel::Exception::create( 'Services::CheckFailed', [ service => $self->{'service'}, message => $DBI::errstr ] );
        }
        alarm(0);
        die Cpanel::Exception::create( 'ConnectionFailed', \%exception_parameters )->to_string() unless $dbh;    ## no extract maketext (variable is metadata; the default message will be used)

        ## check difference between system time and mysql server time
        my @lt       = localtime( time() );
        my $time_str = sprintf( "%04d-%02d-%02d %02d:%02d:%02d", $lt[5] + 1900, $lt[4] + 1, @lt[ 3, 2, 1, 0 ] );
        my $sql      = qq(SELECT HOUR(TIMEDIFF('$time_str', NOW())) as hour_diff);

        my $hour_diff_detected = eval {
            alarm(30);
            my $sth  = $dbh->prepare($sql);
            my $rv   = $sth->execute();
            my $href = $sth->fetchrow_hashref();
            alarm(0);
            $sth->finish();
            $dbh->disconnect();
            $mysqlok = 1;
            $href->{hour_diff} > 0 ? 1 : 0;
        };

        alarm(0);
        if ($hour_diff_detected) {
            die Cpanel::Exception::create( 'Database::ServerTime', \%exception_parameters )->to_string();
        }
    }
You can see how it throws the exception there.

Thoughts? I may have to disable the chkservd monitor on MySQL because of the forced reboots when there are really no issues from what I can tell other than the time zone discrepancy.

If I comment out this code:

Code:
#        if ($hour_diff_detected) { 
#            die Cpanel::Exception::create( 'Database::ServerTime', \%exception_parameters )->to_string(); 
#        }
Then I get this when running the check:

Code:
# /scripts/restartsrv_mysql --check                               
The 'mysql' service passed the check: mysql (/usr/sbin/mysqld) is running as mysql with PID 21990 (systemd+/proc check method).
 
Last edited:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,224
463
Hello @Bigwebmaster,

Could you verify which version of cPanel & WHM is installed on this system? There's a case in version 70 that may address the issue you have reported:

Fixed case CPANEL-18468: Gracefully handle /etc/localtime not being a symlink.

Thank you.
 

Bigwebmaster

Active Member
Dec 3, 2003
32
9
158
We are using: v70.0.34
 

Bigwebmaster

Active Member
Dec 3, 2003
32
9
158
I think a better way to compare MySQL time in this script:

/usr/local/cpanel/Cpanel/ServiceManager/Services/Mysql.pm

is to convert everything into UTC first, and then compare that hours match. A user should be able to set MySQL to use by default a different time zone than the server time. Translated both times are actual equal, and your script should only care that they are using truly the same times.

Cpanel updated to version 70.0.34 this morning (on our server), and that is when all these problems started. I am guessing in versions before this the script that checks times either didn't check the hour before, or something changed there in your logic.
 

Bigwebmaster

Active Member
Dec 3, 2003
32
9
158
Thanks, created a ticket too to see if that can help resolve the problem.
 
  • Like
Reactions: cPanelMichael

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,224
463
Hello,

To update, internal case CPANEL-20407 is open to address an issue in cPanel & WHM version 70 where Chkservd will incorrectly detect MySQL as a down service and subsequently restart it in cases where the default-time-zone option in the /etc/my.cnf file is set to a value that doesn't match the server's time. I'll monitor this case and update this thread with more information as it becomes available. In the meantime, the temporary workarounds are to disable monitoring for MySQL using WHM >> Service Manager or to remove the default-time-zone option in the /etc/my.cnf file.

Thank you.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,224
463
Hello,

To update, this is solved in cPanel & WHM version 70.0.41:

Fixed case CPANEL-20407: Don't restart MySQL if timezone differs from server time.

This version is currently published to the CURRENT build tier, and should reach the RELEASE build tier soon.

Thank you.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,224
463
Hello,

To update, this is solved in cPanel & WHM version 70.0.41:

Fixed case CPANEL-20407: Don't restart MySQL if timezone differs from server time.

This version is currently published to the CURRENT build tier, and should reach the RELEASE build tier soon.

Thank you.
cPanel & WHM version 70.0.41 is now published to the RELEASE build tier.