The cPanel Server operation timed out at cpsrvd.pl line 533

hluoma

Registered
Aug 28, 2018
2
0
1
United States
cPanel Access Level
Root Administrator
My cpanel server has been going down for over the last two days. Sites hosted on the server become unavailable and WHM will not load. I am seeing a 500 error "The cPanel Server operation timed out at cpsrvd.pl line 533" on trying to load the WHM panel and the hosted WP sites are showing Error Establishing Database Connection.

The server is hosted with IBM Softlayer and they were pointing to a brute force attack on xmlrpc.php, however after disabling that file across the whole server we are still seeing the same issues.

grep '' /etc/redhat-release /usr/local/cpanel/version /var/cpanel/envtype ; grep CPANEL= /etc/cpupdate.conf

/etc/redhat-release:CentOS release 6.10 (Final)

/usr/local/cpanel/version:11.56.0.52

/var/cpanel/envtype:xen pv

CPANEL=release

Attaching error_log for today as well
 

Attachments

hluoma

Registered
Aug 28, 2018
2
0
1
United States
cPanel Access Level
Root Administrator
You're really out of date on this server.
You're correct, that is true, unfortunately the previous Admins for the server disabled updates as it was breaking some of the older sites hosted on the server. I inherited the server when my agency bought that company out. For the time being I'm just trying to get it stable so I can keep it up while I move the sites off of it.
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,273
1,282
313
Houston
hi @hluoma


This version of cPanel is End of Life and support for it has ended. With that being said I do understand your position though I'm not aware of sites that would not function because you updated your version of cPanel. Furthermore the specific issue looks to be related to checkservd being unable to restart processes in a timely manner then subsequently running into log write/access errors:


Here tailwatch hangs followed by cpsrvd timeouts:
Code:
[2018-08-28 08:48:12 -0400] info [tailwatchd] chkservd::Hang Notification => [email protected]*****, [email protected]***** via EMAIL [eventimportance => High (1)]
[2018-08-28 08:49:22 -0400] info [cpsrvd] Internal Server Error: "POST /login/?login_only=1 HTTP/1.1" 500 The cPanel Server operation timed out at cpsrvd.pl line 533, <GEN11> line 2.

[2018-08-28 08:51:03 -0400] info [cpsrvd] Internal Server Error: "-" 500 The cPanel Server operation timed out at cpsrvd.pl line 533.

[2018-08-28 08:51:31 -0400] info [cpsrvd] Internal Server Error: "-" 500 The cPanel Server operation timed out at cpsrvd.pl line 533.

[2018-08-28 08:51:33 -0400] info [cpsrvd] Internal Server Error: "-" 500 The cPanel Server operation timed out at cpsrvd.pl line 533.

[2018-08-28 08:56:06 -0400] info [cpsrvd] Internal Server Error: "-" 500 The cPanel Server operation timed out at cpsrvd.pl line 533.
Then right after that:

Code:
[2018-08-28 08:56:44 -0400] warn [cpsrvd] Duplicate logaccess (-):  at /usr/local/cpanel/Cpanel/Server/Logger.pm line 76.
    Cpanel::Server::Logger::logaccess(Cpanel::Server::Logger=HASH(0xb06ef98)) called at /usr/local/cpanel/Cpanel/Server.pm line 638
    Cpanel::Server::body_internal_error(Cpanel::Server=HASH(0xb06f09c), 500, "The cPanel Server operation timed out at cpsrvd.pl line 533.\x{a}") called at /usr/local/cpanel/Cpanel/Server.pm line 571
    Cpanel::Server::internal_error_with_http_status(Cpanel::Server=HASH(0xb06f09c), 500, "The cPanel Server operation timed out at cpsrvd.pl line 533.\x{a}") called at /usr/local/cpanel/Cpanel/Server.pm line 555
    Cpanel::Server::internal_error(Cpanel::Server=HASH(0xb06f09c), "The cPanel Server operation timed out at cpsrvd.pl line 533.\x{a}") called at cpsrvd.pl line 728
    cpanel::cpsrvd::__ANON__(__CPANEL_HIDDEN__) called at /usr/local/cpanel/3rdparty/perl/522/lib/perl5/cpanel_lib/IO/Socket/SSL.pm line 1321
    IO::Socket::SSL::stop_SSL(IO::Socket::SSL=GLOB(0xb071d28), "SSL_fast_shutdown", 1, "_SSL_ioclass_downgrade", 0) called at /usr/local/cpanel/3rdparty/perl/522/lib/perl5/cpanel_lib/IO/Socket/SSL.pm line 1240
    IO::Socket::SSL::close(IO::Socket::SSL=GLOB(0xb071d28)) called at /usr/local/cpanel/Cpanel/Server/Connection.pm line 204
    Cpanel::Server::Connection::shutdown_connection(Cpanel::Server::Connection=HASH(0xb071e7c)) called at /usr/local/cpanel/Cpanel/Server/Connection.pm line 220
    Cpanel::Server::Connection::killconnection(Cpanel::Server::Connection=HASH(0xb071e7c), "ssl redirect", undef) called at /usr/local/cpanel/Cpanel/Server.pm line 218
    Cpanel::Server::killconnection(Cpanel::Server=HASH(0xb06f09c), "ssl redirect") called at cpsrvd.pl line 6756
    cpanel::cpsrvd::ssl_redirect_error(IO::Socket::SSL=GLOB(0xb071d28), "SSL accept attempt failed because of handshake problems\x{a}") called at /usr/local/cpanel/3rdparty/perl/522/lib/perl5/cpanel_lib/IO/Socket/SSL.pm line 1822
    IO::Socket::SSL::fatal_ssl_error(IO::Socket::SSL=GLOB(0xb071d28)) called at /usr/local/cpanel/3rdparty/perl/522/lib/perl5/cpanel_lib/IO/Socket/SSL.pm line 972
    IO::Socket::SSL::accept_SSL(IO::Socket::SSL=GLOB(0xb071d28)) called at /usr/local/cpanel/3rdparty/perl/522/lib/perl5/cpanel_lib/IO/Socket/SSL.pm line 1415
    IO::Socket::SSL::start_SSL(__CPANEL_HIDDEN__, IO::Socket::SSL=GLOB(0xb071d28), __CPANEL_HIDDEN__, IO::Socket::SSL::SSL_Context=HASH(0xb08f64c), __CPANEL_HIDDEN__, CODE(0x878bed0), __CPANEL_HIDDEN__, 1, __CPANEL_HIDDEN__, ...) called at cpsrvd.pl line 930
    cpanel::cpsrvd::handle_one_connection() called at cpsrvd.pl line 841
    cpanel::cpsrvd::script() called at cpsrvd.pl line 334
 at /usr/local/cpanel/Cpanel/Server/Logger.pm line 75.
    Cpanel::Server::Logger::logaccess(Cpanel::Server::Logger=HASH(0xb06ef98)) called at /usr/local/cpanel/Cpanel/Server.pm line 638
    Cpanel::Server::body_internal_error(Cpanel::Server=HASH(0xb06f09c), 500, "The cPanel Server operation timed out at cpsrvd.pl line 533.\x{a}") called at /usr/local/cpanel/Cpanel/Server.pm line 571
    Cpanel::Server::internal_error_with_http_status(Cpanel::Server=HASH(0xb06f09c), 500, "The cPanel Server operation timed out at cpsrvd.pl line 533.\x{a}") called at /usr/local/cpanel/Cpanel/Server.pm line 555
    Cpanel::Server::internal_error(Cpanel::Server=HASH(0xb06f09c), "The cPanel Server operation timed out at cpsrvd.pl line 533.\x{a}") called at cpsrvd.pl line 728
    cpanel::cpsrvd::__ANON__(__CPANEL_HIDDEN__) called at /usr/local/cpanel/3rdparty/perl/522/lib/perl5/cpanel_lib/IO/Socket/SSL.pm line 1321
    IO::Socket::SSL::stop_SSL(IO::Socket::SSL=GLOB(0xb071d28), "SSL_fast_shutdown", 1, "_SSL_ioclass_downgrade", 0) called at /usr/local/cpanel/3rdparty/perl/522/lib/perl5/cpanel_lib/IO/Socket/SSL.pm line 1240
    IO::Socket::SSL::close(IO::Socket::SSL=GLOB(0xb071d28)) called at /usr/local/cpanel/Cpanel/Server/Connection.pm line 204
    Cpanel::Server::Connection::shutdown_connection(Cpanel::Server::Connection=HASH(0xb071e7c)) called at /usr/local/cpanel/Cpanel/Server/Connection.pm line 220
    Cpanel::Server::Connection::killconnection(Cpanel::Server::Connection=HASH(0xb071e7c), "ssl redirect", undef) called at /usr/local/cpanel/Cpanel/Server.pm line 218
    Cpanel::Server::killconnection(Cpanel::Server=HASH(0xb06f09c), "ssl redirect") called at cpsrvd.pl line 6756
    cpanel::cpsrvd::ssl_redirect_error(IO::Socket::SSL=GLOB(0xb071d28), "SSL accept attempt failed because of handshake problems\x{a}") called at /usr/local/cpanel/3rdparty/perl/522/lib/perl5/cpanel_lib/IO/Socket/SSL.pm line 1822
    IO::Socket::SSL::fatal_ssl_error(IO::Socket::SSL=GLOB(0xb071d28)) called at /usr/local/cpanel/3rdparty/perl/522/lib/perl5/cpanel_lib/IO/Socket/SSL.pm line 972
    IO::Socket::SSL::accept_SSL(IO::Socket::SSL=GLOB(0xb071d28)) called at /usr/local/cpanel/3rdparty/perl/522/lib/perl5/cpanel_lib/IO/Socket/SSL.pm line 1415
    IO::Socket::SSL::start_SSL(__CPANEL_HIDDEN__, IO::Socket::SSL=GLOB(0xb071d28), __CPANEL_HIDDEN__, IO::Socket::SSL::SSL_Context=HASH(0xb08f64c), __CPANEL_HIDDEN__, CODE(0x878bed0), __CPANEL_HIDDEN__, 1, __CPANEL_HIDDEN__, ...) called at cpsrvd.pl line 930
    cpanel::cpsrvd::handle_one_connection() called at cpsrvd.pl line 841
    cpanel::cpsrvd::script() called at cpsrvd.pl line 334
You might try manually restarting chkservd, cpsrvd and tailwatchd to ensure that they restart normally but I cannot guarantee this will resolve this issue.

To restart do the following:
Code:
/scripts/restartsrv_cpsrvd
/scripts/restartsrv_tailwatchd
/scripts/restartsrv_chkservd