The cPanel Server operation timed out

Topotyn

Member
Sep 19, 2019
6
1
1
Ukraine
cPanel Access Level
Root Administrator
Good afternoon. I recently had a problem. Server with whm freezes, upgrade from version 82.0.14 to 82.0.15 helped. I see errors in the log file, but I can not figure out what happened. Help me please. If you need more information, ask.

Code:
[2019-09-16 14:16:04 -0400] info [cpsrvd] version 11.82.0.14 online
[2019-09-16 14:29:08 -0400] warn [cpsrvd] Previous check_sessions still running with pid: 6061.  Possible cpsrvd denial of service attack in progress. at cpsrvd.pl line 1150.
    cpanel::cpsrvd::script() called at cpsrvd.pl line 427
[2019-09-16 14:30:29 -0400] warn [cpses_tool] Error while connecting to MySQL: (XID g6mn6e) The system failed to connect to the “MySQL” database “mysql” because of an error: CR_CONNECTION_ERROR (Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)) at /usr/local/cpanel/Cpanel/DBAdmin.pm line 187, <STDIN> line 1.
    Cpanel::DBAdmin::_log_error_and_output_return(Cpanel::Mysql=HASH(0x285bb70), Cpanel::LocaleString=ARRAY(0x294c400)) called at /usr/local/cpanel/Cpanel/Mysql/Basic.pm line 391
    Cpanel::Mysql::Basic::_get_dbh(Cpanel::Mysql=HASH(0x285bb70), __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, 3306) called at /usr/local/cpanel/Cpanel/Mysql/Basic.pm line 79
    Cpanel::Mysql::Basic::new("Cpanel::Mysql", HASH(0x2851318)) called at bin/cpses_tool.pl line 274
    bin::cpses_tool::_get_db_objects("root") called at bin/cpses_tool.pl line 199
    bin::cpses_tool::action_CLEANUPSESSIONS("bin::cpses_tool", HASH(0x28482a0)) called at bin/cpses_tool.pl line 75
    bin::cpses_tool::process_request(HASH(0x28482a0)) called at bin/cpses_tool.pl line 60
    bin::cpses_tool::script("bin::cpses_tool") called at bin/cpses_tool.pl line 33
Error while connecting to MySQL: (XID g6mn6e) The system failed to connect to the “MySQL” database “mysql” because of an error: CR_CONNECTION_ERROR (Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)) at /usr/local/cpanel/Cpanel/Mysql/Basic.pm line 391, <STDIN> line 1.
[2019-09-16 14:33:57 -0400] warn [tailwatchd] [7128] TIME 1568658808 UID 0 clobbered stale lock file “/var/run/chkservd.checking.lock” (NAME “tailwatchd - chkservd”, UID 0) written by PID 5613 at 1568657745 at /usr/local/cpanel/Cpanel/SafeFile.pm line 931.
    Cpanel::SafeFile::_lock_wait("/var/run/chkservd.checking", Cpanel::SafeFileLock=ARRAY(0x19572a8), "/var/run/chkservd.checking.lock") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 350
    Cpanel::SafeFile::_safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 294
    Cpanel::SafeFile::safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 252
    Cpanel::TailWatch::ChkServd::_run_checks(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568658582) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 208
    Cpanel::TailWatch::ChkServd::run(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568658582) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 754
    Cpanel::TailWatch::tail_logs(Cpanel::TailWatch=HASH(0x9897a0)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 657
    eval {...} called at /usr/local/cpanel/Cpanel/TailWatch.pm line 625
    Cpanel::TailWatch::run(Cpanel::TailWatch=HASH(0x9897a0), 0, "resume", 1) called at libexec/tailwatch/tailwatchd.pl line 62
    main::run("--resume") called at libexec/tailwatch/tailwatchd.pl line 35
[2019-09-16 14:34:28 -0400] warn [pureauth] Failed to call hulk pre_login for [email protected] (system) at bin/pureauth.pl line 136.
[2019-09-16 14:43:29 -0400] warn [tailwatchd] [7749] TIME 1568659408 UID 0 clobbered stale lock file “/var/run/chkservd.checking.lock” (NAME “tailwatchd - chkservd”, UID 0) written by PID 7128 at 1568658782 at /usr/local/cpanel/Cpanel/SafeFile.pm line 931.
    Cpanel::SafeFile::_lock_wait("/var/run/chkservd.checking", Cpanel::SafeFileLock=ARRAY(0x194a4e8), "/var/run/chkservd.checking.lock") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 350
    Cpanel::SafeFile::_safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 294
    Cpanel::SafeFile::safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 252
    Cpanel::TailWatch::ChkServd::_run_checks(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568659321) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 208
    Cpanel::TailWatch::ChkServd::run(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568659321) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 754
    Cpanel::TailWatch::tail_logs(Cpanel::TailWatch=HASH(0x9897a0)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 657
    eval {...} called at /usr/local/cpanel/Cpanel/TailWatch.pm line 625
    Cpanel::TailWatch::run(Cpanel::TailWatch=HASH(0x9897a0), 0, "resume", 1) called at libexec/tailwatch/tailwatchd.pl line 62
    main::run("--resume") called at libexec/tailwatch/tailwatchd.pl line 35
[2019-09-16 14:43:43 -0400] warn [cpsrvd] Previous check_sessions still running with pid: 7452.  Possible cpsrvd denial of service attack in progress. at cpsrvd.pl line 1150.
    cpanel::cpsrvd::script() called at cpsrvd.pl line 427
[2019-09-16 14:43:57 -0400] warn [cpses_tool] Error while connecting to MySQL: (XID r8ezqe) The system failed to connect to the “MySQL” database “mysql” because of an error: CR_CONNECTION_ERROR (Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)) at /usr/local/cpanel/Cpanel/DBAdmin.pm line 187, <STDIN> line 1.
    Cpanel::DBAdmin::_log_error_and_output_return(Cpanel::Mysql=HASH(0x1880b70), Cpanel::LocaleString=ARRAY(0x1971140)) called at /usr/local/cpanel/Cpanel/Mysql/Basic.pm line 391
    Cpanel::Mysql::Basic::_get_dbh(Cpanel::Mysql=HASH(0x1880b70), __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, 3306) called at /usr/local/cpanel/Cpanel/Mysql/Basic.pm line 79
    Cpanel::Mysql::Basic::new("Cpanel::Mysql", HASH(0x1876318)) called at bin/cpses_tool.pl line 274
    bin::cpses_tool::_get_db_objects("root") called at bin/cpses_tool.pl line 199
    bin::cpses_tool::action_CLEANUPSESSIONS("bin::cpses_tool", HASH(0x186d2a0)) called at bin/cpses_tool.pl line 75
    bin::cpses_tool::process_request(HASH(0x186d2a0)) called at bin/cpses_tool.pl line 60
    bin::cpses_tool::script("bin::cpses_tool") called at bin/cpses_tool.pl line 33
Error while connecting to MySQL: (XID r8ezqe) The system failed to connect to the “MySQL” database “mysql” because of an error: CR_CONNECTION_ERROR (Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)) at /usr/local/cpanel/Cpanel/Mysql/Basic.pm line 391, <STDIN> line 1.
[2019-09-16 14:44:50 -0400] info [tailwatchd] chkservd::OOM Notification => [email protected]***** via EMAIL [eventimportance => High (1)]
[2019-09-16 14:48:00 -0400] info [tailwatchd] chkservd::Hang Notification => [email protected]***** via EMAIL [eventimportance => High (1)]
[2019-09-16 14:48:00 -0400] info [tailwatchd] chkservd::Hang Notification => [email protected]***** via EMAIL [eventimportance => High (1)]
==> cpsrvd 11.82.0.14 started
==> cpsrvd: loading security policy....Done
==> cpsrvd: Setting up SSL support ... Done
==> cpsrvd: transferred port bindings: 10,11,12,3,4,5,6,7,8,9
==> cpsrvd: bound to ports
[2019-09-16 14:49:31 -0400] warn [tailwatchd] Cpanel::Exception::SMTP/(XID gcve69) The system failed to authenticate with the [asis,SMTP] server “127.0.0.1” because of an error: [Net::SMTP] Timeout
 at /usr/local/cpanel/Cpanel/SMTP.pm line 164.
    Cpanel::SMTP::auth(Cpanel::SMTP=ARRAY(0x2cca218), __CPANEL_HIDDEN__, __CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/SMTP/Singleton.pm line 103
    Cpanel::SMTP::Singleton::localhost(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 120
    Cpanel::Email::Send::__ANON__(Cpanel::Exception::SMTP=HASH(0x31abf88)) called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 118
    Try::Tiny::try(CODE(0x2cc1ba0), Try::Tiny::Catch=REF(0x2ca1980)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 122
    Cpanel::Email::Send::_email_message_using_smtp(HASH(0x2c942f0), HASH(0x2cb1f30)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 86
    Cpanel::Email::Send::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x2ad8e18), Try::Tiny::Catch=REF(0x2ac67a0)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 101
    Cpanel::Email::Send::email_message(HASH(0x2c942f0), HASH(0x2cb1f30)) called at /usr/local/cpanel/Cpanel/iContact/Provider.pm line 71
    Cpanel::iContact::Provider::email_message(Cpanel::iContact::Provider::Email=HASH(0x2c7c1e8), "to", ARRAY(0x2ace150), "x_headers", HASH(0x2aec130), "subject", "[name_server] Out of memory: \x{e2}\x{9a}\x{a0} The process \x{e2}\x{80}\x{9c}cP"..., "from", ...) called at /usr/local/cpanel/Cpanel/iContact/Provider/Email.pm line 32
    Cpanel::iContact::Provider::Email::send(Cpanel::iContact::Provider::Email=HASH(0x2c7c1e8)) called at /usr/local/cpanel/Cpanel/iContact.pm line 566
    Cpanel::iContact::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x2ad8710), Try::Tiny::Catch=REF(0x2acfe38)) called at /usr/local/cpanel/Cpanel/iContact.pm line 570
    Cpanel::iContact::_send_notifications(HASH(0x1e3e868), HASH(0x1ecacd0), ARRAY(0x1c1aa50)) called at /usr/local/cpanel/Cpanel/iContact.pm line 523
    Cpanel::iContact::icontact("subject", "Out of memory: \x{e2}\x{9a}\x{a0} The process \x{e2}\x{80}\x{9c}cPhulkd - dbpro\x{e2}\x{80}\x{9d} was term"..., "html_related", ARRAY(0x1cf8a38), "x_headers", HASH(0x2aec130), "application", "chkservd", ...) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 567
    Cpanel::iContact::Class::_todo_inside_daemon(Cpanel::iContact::Class::chkservd::OOM=HASH(0x1c7f0f0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 371
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 89
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x1c7f990), Try::Tiny::Catch=REF(0x1b71df0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 94
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    eval {...} called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    Cpanel::ForkAsync::do_in_child(CODE(0x1c7f318)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 96
    Cpanel::iContact::Class::_do_in_daemon(CODE(0x1c7f198)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 373
    Cpanel::iContact::Class::send(Cpanel::iContact::Class::chkservd::OOM=HASH(0x1c7f0f0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 350
    Cpanel::iContact::Class::new("Cpanel::iContact::Class::chkservd::OOM", "process_killed", 1, "seconds_since_boot", 1599891.117932, "uid", 0, "pid", ...) called at /usr/local/cpanel/Cpanel/Notify.pm line 77
    Cpanel::Notify::__ANON__() called at /usr/local/cpanel/Cpanel/Notify.pm line 150
    Cpanel::Notify::_notification_backend("oomcheck", "0,cPhulkd - dbpro", 10368000, CODE(0x1c5fac8)) called at /usr/local/cpanel/Cpanel/Notify.pm line 79
    Cpanel::Notify::notification_class("application", "oomcheck", "interval", 10368000, "status", "0,cPhulkd - dbpro", "class", "chkservd::OOM", ...) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 724
    Cpanel::TailWatch::ChkServd::_oomcheck(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0)) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 389
    Cpanel::TailWatch::ChkServd::_check_services(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0)) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 276
    Cpanel::TailWatch::ChkServd::_run_checks(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568657692) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 208
    Cpanel::TailWatch::ChkServd::run(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568657692) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 754
    Cpanel::TailWatch::tail_logs(Cpanel::TailWatch=HASH(0x9897a0)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 657
    eval {...} called at /usr/local/cpanel/Cpanel/TailWatch.pm line 625
    Cpanel::TailWatch::run(Cpanel::TailWatch=HASH(0x9897a0), 0, "resume", 1) called at libexec/tailwatch/tailwatchd.pl line 62
    main::run("--resume") called at libexec/tailwatch/tailwatchd.pl line 35
[2019-09-16 14:53:40 -0400] info [cpsrvd] version 11.82.0.14 online
[2019-09-16 14:53:50 -0400] info [tailwatchd] chkservd::OOM Notification => [email protected]***** via EMAIL [eventimportance => High (1)]
[2019-09-16 14:53:50 -0400] info [tailwatchd] chkservd::OOM Notification => [email protected]***** via EMAIL [eventimportance => High (1)]
[2019-09-16 14:53:53 -0400] warn [tailwatchd] Cpanel::Exception::SMTP/(XID vzucb6) The system failed to authenticate with the [asis,SMTP] server “127.0.0.1” because of an error: [Net::SMTP] Timeout
 at /usr/local/cpanel/Cpanel/SMTP.pm line 164.
    Cpanel::SMTP::auth(Cpanel::SMTP=ARRAY(0x2b6f690), __CPANEL_HIDDEN__, __CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/SMTP/Singleton.pm line 103
    Cpanel::SMTP::Singleton::localhost(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 120
    Cpanel::Email::Send::__ANON__(Cpanel::Exception::SMTP=HASH(0x30b72c0)) called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 118
    Try::Tiny::try(CODE(0x2b9bff0), Try::Tiny::Catch=REF(0x2b9bfa8)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 122
    Cpanel::Email::Send::_email_message_using_smtp(HASH(0x2a2fb68), HASH(0x292d6c8)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 86
    Cpanel::Email::Send::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x2b6f7b0), Try::Tiny::Catch=REF(0x29bd988)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 101
    Cpanel::Email::Send::email_message(HASH(0x2a2fb68), HASH(0x292d6c8)) called at /usr/local/cpanel/Cpanel/iContact/Provider.pm line 71
    Cpanel::iContact::Provider::email_message(Cpanel::iContact::Provider::Email=HASH(0x2a2f9b8), "im_subject", "[name_server] HANG: ???: chkservd (ip_address)", "application", "chkservd", "event_name", "Hang", "im_message", ...) called at /usr/local/cpanel/Cpanel/iContact/Provider/Email.pm line 32
    Cpanel::iContact::Provider::Email::send(Cpanel::iContact::Provider::Email=HASH(0x2a2f9b8)) called at /usr/local/cpanel/Cpanel/iContact.pm line 566
    Cpanel::iContact::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x25c2e78), Try::Tiny::Catch=REF(0x2930b90)) called at /usr/local/cpanel/Cpanel/iContact.pm line 570
    Cpanel::iContact::_send_notifications(HASH(0x1bc0250), HASH(0x1cfd9a8), ARRAY(0x1a50880)) called at /usr/local/cpanel/Cpanel/iContact.pm line 523
    Cpanel::iContact::icontact("subject", "HANG: \x{e2}\x{9a}\x{a0}: chkservd (ip_address)", "html_related", ARRAY(0x1a14940), "x_headers", HASH(0x29379e8), "plaintext_message", "The chkservd process has become non-responsive.\x{a}\x{a}The chkservd"..., ...) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 567
    Cpanel::iContact::Class::_todo_inside_daemon(Cpanel::iContact::Class::chkservd::Hang=HASH(0x19646f0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 371
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 89
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x1b184d0), Try::Tiny::Catch=REF(0x1775678)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 94
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    eval {...} called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    Cpanel::ForkAsync::do_in_child(CODE(0x1a69e40)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 96
    Cpanel::iContact::Class::_do_in_daemon(CODE(0x19c95d0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 373
    Cpanel::iContact::Class::send(Cpanel::iContact::Class::chkservd::Hang=HASH(0x19646f0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 350
    Cpanel::iContact::Class::new("Cpanel::iContact::Class::chkservd::Hang", "child_run_time", 739, "child_pid", 7128, "check_interval", 300, "from", ...) called at /usr/local/cpanel/Cpanel/iContact/Class/chkservd/Hang.pm line 50
    Cpanel::iContact::Class::chkservd::Hang::new("Cpanel::iContact::Class::chkservd::Hang", "child_run_time", 739, "child_pid", 7128, "check_interval", 300, "body_format", ...) called at /usr/local/cpanel/Cpanel/Notify.pm line 77
    Cpanel::Notify::__ANON__() called at /usr/local/cpanel/Cpanel/Notify.pm line 150
    Cpanel::Notify::_notification_backend("chkservd::Hang", "hang", 1, CODE(0x128fb10)) called at /usr/local/cpanel/Cpanel/Notify.pm line 79
    Cpanel::Notify::notification_class("interval", 1, "status", "hang", "class", "chkservd::Hang", "application", "chkservd::Hang", ...) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 225
    Cpanel::TailWatch::ChkServd::_send_hang_notification(HASH(0x18c7bb8), "html") called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 190
    eval {...} called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 190
    Cpanel::TailWatch::ChkServd::__ANON__() called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    eval {...} called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    Cpanel::ForkAsync::do_in_child(CODE(0xbf4b28)) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 200
    Cpanel::TailWatch::ChkServd::run(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568659321) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 754
    Cpanel::TailWatch::tail_logs(Cpanel::TailWatch=HASH(0x9897a0)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 657
    eval {...} called at /usr/local/cpanel/Cpanel/TailWatch.pm line 625
    Cpanel::TailWatch::run(Cpanel::TailWatch=HASH(0x9897a0), 0, "resume", 1) called at libexec/tailwatch/tailwatchd.pl line 62
    main::run("--resume") called at libexec/tailwatch/tailwatchd.pl line 35
[2019-09-16 14:53:53 -0400] warn [tailwatchd] Cpanel::Exception::SMTP/(XID yrkpuy) The system failed to authenticate with the [asis,SMTP] server “127.0.0.1” because of an error: [Net::SMTP] Timeout
 at /usr/local/cpanel/Cpanel/SMTP.pm line 164.
    Cpanel::SMTP::auth(Cpanel::SMTP=ARRAY(0x2b3b7c8), __CPANEL_HIDDEN__, __CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/SMTP/Singleton.pm line 103
    Cpanel::SMTP::Singleton::localhost(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 120
    Cpanel::Email::Send::__ANON__(Cpanel::Exception::SMTP=HASH(0x30952e8)) called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 118
    Try::Tiny::try(CODE(0x2b896e0), Try::Tiny::Catch=REF(0x2b89698)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 122
    Cpanel::Email::Send::_email_message_using_smtp(HASH(0x2ad4d88), HASH(0x2918140)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 86
    Cpanel::Email::Send::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x2b5b470), Try::Tiny::Catch=REF(0x291d268)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 101
    Cpanel::Email::Send::email_message(HASH(0x2ad4d88), HASH(0x2918140)) called at /usr/local/cpanel/Cpanel/iContact/Provider.pm line 71
    Cpanel::iContact::Provider::email_message(Cpanel::iContact::Provider::Email=HASH(0x2abc570), "from", "\"cPanel Monitoring\" <cpanel\@name_server>", "subject", "[name_server] HANG: \x{e2}\x{9a}\x{a0}: chkservd (ip_address)", "to", ARRAY(0x2917f30), "x_headers", ...) called at /usr/local/cpanel/Cpanel/iContact/Provider/Email.pm line 32
    Cpanel::iContact::Provider::Email::send(Cpanel::iContact::Provider::Email=HASH(0x2abc570)) called at /usr/local/cpanel/Cpanel/iContact.pm line 566
    Cpanel::iContact::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x290bb48), Try::Tiny::Catch=REF(0x2917d50)) called at /usr/local/cpanel/Cpanel/iContact.pm line 570
    Cpanel::iContact::_send_notifications(HASH(0x1bac510), HASH(0x1ce9648), ARRAY(0x19a9370)) called at /usr/local/cpanel/Cpanel/iContact.pm line 523
    Cpanel::iContact::icontact("subject", "HANG: \x{e2}\x{9a}\x{a0}: chkservd (ip_address)", "html_related", ARRAY(0x1a00030), "x_headers", HASH(0x2967c08), "im_message", "The chkservd process has become non-responsive.\x{a}\x{a}The chkservd"..., ...) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 567
    Cpanel::iContact::Class::_todo_inside_daemon(Cpanel::iContact::Class::chkservd::Hang=HASH(0x194fc00)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 371
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 89
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x1b03640), Try::Tiny::Catch=REF(0x1775678)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 94
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    eval {...} called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    Cpanel::ForkAsync::do_in_child(CODE(0x193f1b0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 96
    Cpanel::iContact::Class::_do_in_daemon(CODE(0x1aa6ab0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 373
    Cpanel::iContact::Class::send(Cpanel::iContact::Class::chkservd::Hang=HASH(0x194fc00)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 350
    Cpanel::iContact::Class::new("Cpanel::iContact::Class::chkservd::Hang", "child_run_time", 890, "child_pid", 5613, "body_format", "html", "from", ...) called at /usr/local/cpanel/Cpanel/iContact/Class/chkservd/Hang.pm line 50
    Cpanel::iContact::Class::chkservd::Hang::new("Cpanel::iContact::Class::chkservd::Hang", "body_format", "html", "from", "cPanel Monitoring", "check_interval", 300, "child_pid", ...) called at /usr/local/cpanel/Cpanel/Notify.pm line 77
    Cpanel::Notify::__ANON__() called at /usr/local/cpanel/Cpanel/Notify.pm line 150
    Cpanel::Notify::_notification_backend("chkservd::Hang", "hang", 1, CODE(0x18398e8)) called at /usr/local/cpanel/Cpanel/Notify.pm line 79
    Cpanel::Notify::notification_class("interval", 1, "status", "hang", "class", "chkservd::Hang", "application", "chkservd::Hang", ...) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 225
    Cpanel::TailWatch::ChkServd::_send_hang_notification(HASH(0x18c79f0), "html") called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 190
    eval {...} called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 190
    Cpanel::TailWatch::ChkServd::__ANON__() called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    eval {...} called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    Cpanel::ForkAsync::do_in_child(CODE(0x1843e18)) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 200
    Cpanel::TailWatch::ChkServd::run(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568658582) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 754
    Cpanel::TailWatch::tail_logs(Cpanel::TailWatch=HASH(0x9897a0)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 657
    eval {...} called at /usr/local/cpanel/Cpanel/TailWatch.pm line 625
    Cpanel::TailWatch::run(Cpanel::TailWatch=HASH(0x9897a0), 0, "resume", 1) called at libexec/tailwatch/tailwatchd.pl line 62
    main::run("--resume") called at libexec/tailwatch/tailwatchd.pl line 35
[2019-09-16 14:53:53 -0400] warn [tailwatchd] [8532] TIME 1568660029 UID 0 clobbered stale lock file “/var/run/chkservd.checking.lock” (NAME “tailwatchd - chkservd”, UID 0) written by PID 7749 at 1568659406 at /usr/local/cpanel/Cpanel/SafeFile.pm line 931.
    Cpanel::SafeFile::_lock_wait("/var/run/chkservd.checking", Cpanel::SafeFileLock=ARRAY(0x197ade8), "/var/run/chkservd.checking.lock") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 350
    Cpanel::SafeFile::_safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 294
    Cpanel::SafeFile::safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 252
    Cpanel::TailWatch::ChkServd::_run_checks(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568659925) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 208
    Cpanel::TailWatch::ChkServd::run(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568659925) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 754
    Cpanel::TailWatch::tail_logs(Cpanel::TailWatch=HASH(0x9897a0)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 657
    eval {...} called at /usr/local/cpanel/Cpanel/TailWatch.pm line 625
    Cpanel::TailWatch::run(Cpanel::TailWatch=HASH(0x9897a0), 0, "resume", 1) called at libexec/tailwatch/tailwatchd.pl line 62
    main::run("--resume") called at libexec/tailwatch/tailwatchd.pl line 35
[2019-09-16 14:55:39 -0400] warn [tailwatchd] Cpanel::Exception::SMTP/(XID k4chcs) The system failed to complete its data transmission to the [asis,SMTP] server “127.0.0.1” because of an error: [Net::SMTP] Timeout
 at /usr/local/cpanel/Cpanel/SMTP.pm line 146.
    Cpanel::SMTP::dataend(Cpanel::SMTP=ARRAY(0x2d388d0)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 161
    Cpanel::Email::Send::_email_message_using_smtp(HASH(0x2cf7eb0), HASH(0x2b4a878)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 86
    Cpanel::Email::Send::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x2cf7fa0), Try::Tiny::Catch=REF(0x2b4dda8)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 101
    Cpanel::Email::Send::email_message(HASH(0x2cf7eb0), HASH(0x2b4a878)) called at /usr/local/cpanel/Cpanel/iContact/Provider.pm line 71
    Cpanel::iContact::Provider::email_message(Cpanel::iContact::Provider::Email=HASH(0x2cf7d30), "im_subject", "[name_server] Out of memory: ??? The process \"java"..., "event_name", "OOM", "application", "chkservd", "html_body", ...) called at /usr/local/cpanel/Cpanel/iContact/Provider/Email.pm line 32
    Cpanel::iContact::Provider::Email::send(Cpanel::iContact::Provider::Email=HASH(0x2cf7d30)) called at /usr/local/cpanel/Cpanel/iContact.pm line 566
    Cpanel::iContact::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x2b4b760), Try::Tiny::Catch=REF(0x2b4b790)) called at /usr/local/cpanel/Cpanel/iContact.pm line 570
    Cpanel::iContact::_send_notifications(HASH(0x1eaa3b0), HASH(0x1f399d8), ARRAY(0x1c70128)) called at /usr/local/cpanel/Cpanel/iContact.pm line 523
    Cpanel::iContact::icontact("subject", "Out of memory: \x{e2}\x{9a}\x{a0} The process \x{e2}\x{80}\x{9c}java\x{e2}\x{80}\x{9d} was terminated beca"..., "html_related", ARRAY(0x1d19430), "x_headers", HASH(0x2bd76c0), "from", "cPanel Monitoring", ...) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 567
    Cpanel::iContact::Class::_todo_inside_daemon(Cpanel::iContact::Class::chkservd::OOM=HASH(0x1dc4cd0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 371
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 89
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x1dc5510), Try::Tiny::Catch=REF(0x1bc0cb0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 94
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    eval {...} called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    Cpanel::ForkAsync::do_in_child(CODE(0x1dd3dd8)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 96
    Cpanel::iContact::Class::_do_in_daemon(CODE(0x1dc4dd8)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 373
    Cpanel::iContact::Class::send(Cpanel::iContact::Class::chkservd::OOM=HASH(0x1dc4cd0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 350
    Cpanel::iContact::Class::new("Cpanel::iContact::Class::chkservd::OOM", "attach_files", ARRAY(0x1c70128), "score", 71, "file_rss", "0kB", "time", ...) called at /usr/local/cpanel/Cpanel/Notify.pm line 77
    Cpanel::Notify::__ANON__() called at /usr/local/cpanel/Cpanel/Notify.pm line 150
    Cpanel::Notify::_notification_backend("oomcheck", "986,java", 10368000, CODE(0x1cea1c0)) called at /usr/local/cpanel/Cpanel/Notify.pm line 79
    Cpanel::Notify::notification_class("application", "oomcheck", "interval", 10368000, "status", "986,java", "class", "chkservd::OOM", ...) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 724
    Cpanel::TailWatch::ChkServd::_oomcheck(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0)) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 389
    Cpanel::TailWatch::ChkServd::_check_services(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0)) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 276
    Cpanel::TailWatch::ChkServd::_run_checks(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568659321) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 208
    Cpanel::TailWatch::ChkServd::run(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568659321) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 754
    Cpanel::TailWatch::tail_logs(Cpanel::TailWatch=HASH(0x9897a0)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 657
    eval {...} called at /usr/local/cpanel/Cpanel/TailWatch.pm line 625
    Cpanel::TailWatch::run(Cpanel::TailWatch=HASH(0x9897a0), 0, "resume", 1) called at libexec/tailwatch/tailwatchd.pl line 62
    main::run("--resume") called at libexec/tailwatch/tailwatchd.pl line 35
[2019-09-16 14:55:43 -0400] warn [tailwatchd] Cpanel::Exception::SMTP/(XID aegdms) The system failed to complete its data transmission to the [asis,SMTP] server “127.0.0.1” because of an error: [Net::SMTP] Timeout
 at /usr/local/cpanel/Cpanel/SMTP.pm line 146.
    Cpanel::SMTP::dataend(Cpanel::SMTP=ARRAY(0x2d3d4f8)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 161
    Cpanel::Email::Send::_email_message_using_smtp(HASH(0x2cfcc98), HASH(0x2b4f538)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 86
    Cpanel::Email::Send::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x2bf8b48), Try::Tiny::Catch=REF(0x2b52c18)) called at /usr/local/cpanel/Cpanel/Email/Send.pm line 101
    Cpanel::Email::Send::email_message(HASH(0x2cfcc98), HASH(0x2b4f538)) called at /usr/local/cpanel/Cpanel/iContact/Provider.pm line 71
    Cpanel::iContact::Provider::email_message(Cpanel::iContact::Provider::Email=HASH(0x2cfcb18), "im_subject", "[name_server] Out of memory: ??? The process \"mysq"..., "application", "chkservd", "event_name", "OOM", "html_body", ...) called at /usr/local/cpanel/Cpanel/iContact/Provider/Email.pm line 32
    Cpanel::iContact::Provider::Email::send(Cpanel::iContact::Provider::Email=HASH(0x2cfcb18)) called at /usr/local/cpanel/Cpanel/iContact.pm line 566
    Cpanel::iContact::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x2b52ac8), Try::Tiny::Catch=REF(0x2b52948)) called at /usr/local/cpanel/Cpanel/iContact.pm line 570
    Cpanel::iContact::_send_notifications(HASH(0x1eae0b8), HASH(0x1f3f928), ARRAY(0x1c70128)) called at /usr/local/cpanel/Cpanel/iContact.pm line 523
    Cpanel::iContact::icontact("subject", "Out of memory: \x{e2}\x{9a}\x{a0} The process \x{e2}\x{80}\x{9c}mysqld\x{e2}\x{80}\x{9d} was terminated be"..., "html_related", ARRAY(0x1d19430), "x_headers", HASH(0x2bdc150), "message", "<body style=\"background:#F4F4F4\">\x{a}    <div style=\"margin:0;pa"..., ...) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 567
    Cpanel::iContact::Class::_todo_inside_daemon(Cpanel::iContact::Class::chkservd::OOM=HASH(0x1ce4a90)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 371
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 89
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97
    eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88
    Try::Tiny::try(CODE(0x1cfc4e0), Try::Tiny::Catch=REF(0x1bc0cb0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 94
    Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    eval {...} called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 68
    Cpanel::ForkAsync::do_in_child(CODE(0x1ddf0b0)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 96
    Cpanel::iContact::Class::_do_in_daemon(CODE(0x1c80798)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 373
    Cpanel::iContact::Class::send(Cpanel::iContact::Class::chkservd::OOM=HASH(0x1ce4a90)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 350
    Cpanel::iContact::Class::new("Cpanel::iContact::Class::chkservd::OOM", "pid", 3472, "anon_rss", "75028kB", "proc_name", "mysqld", "user", ...) called at /usr/local/cpanel/Cpanel/Notify.pm line 77
    Cpanel::Notify::__ANON__() called at /usr/local/cpanel/Cpanel/Notify.pm line 150
    Cpanel::Notify::_notification_backend("oomcheck", "27,mysqld", 10368000, CODE(0x1ce9ef0)) called at /usr/local/cpanel/Cpanel/Notify.pm line 79
    Cpanel::Notify::notification_class("application", "oomcheck", "interval", 10368000, "status", "27,mysqld", "class", "chkservd::OOM", ...) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 724
    Cpanel::TailWatch::ChkServd::_oomcheck(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0)) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 389
    Cpanel::TailWatch::ChkServd::_check_services(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0)) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 276
    Cpanel::TailWatch::ChkServd::_run_checks(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568659321) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 208
    Cpanel::TailWatch::ChkServd::run(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568659321) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 754
    Cpanel::TailWatch::tail_logs(Cpanel::TailWatch=HASH(0x9897a0)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 657
    eval {...} called at /usr/local/cpanel/Cpanel/TailWatch.pm line 625
    Cpanel::TailWatch::run(Cpanel::TailWatch=HASH(0x9897a0), 0, "resume", 1) called at libexec/tailwatch/tailwatchd.pl line 62
    main::run("--resume") called at libexec/tailwatch/tailwatchd.pl line 35
[2019-09-16 15:00:19 -0400] info [tailwatchd] chkservd::Hang Notification => [email protected]***** via EMAIL [eventimportance => High (1)]
[2019-09-16 15:01:33 -0400] warn [cpses_tool] Error while connecting to MySQL: (XID q6ga4e) The system failed to connect to the “MySQL” database “mysql” because of an error: CR_CONNECTION_ERROR (Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)) at /usr/local/cpanel/Cpanel/DBAdmin.pm line 187, <STDIN> line 1.
    Cpanel::DBAdmin::_log_error_and_output_return(Cpanel::Mysql=HASH(0x14c2b70), Cpanel::LocaleString=ARRAY(0x15b7eb0)) called at /usr/local/cpanel/Cpanel/Mysql/Basic.pm line 391
    Cpanel::Mysql::Basic::_get_dbh(Cpanel::Mysql=HASH(0x14c2b70), __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, 3306) called at /usr/local/cpanel/Cpanel/Mysql/Basic.pm line 79
    Cpanel::Mysql::Basic::new("Cpanel::Mysql", HASH(0x14b8318)) called at bin/cpses_tool.pl line 274
    bin::cpses_tool::_get_db_objects("root") called at bin/cpses_tool.pl line 199
    bin::cpses_tool::action_CLEANUPSESSIONS("bin::cpses_tool", HASH(0x14af2a0)) called at bin/cpses_tool.pl line 75
    bin::cpses_tool::process_request(HASH(0x14af2a0)) called at bin/cpses_tool.pl line 60
    bin::cpses_tool::script("bin::cpses_tool") called at bin/cpses_tool.pl line 33
Error while connecting to MySQL: (XID q6ga4e) The system failed to connect to the “MySQL” database “mysql” because of an error: CR_CONNECTION_ERROR (Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)) at /usr/local/cpanel/Cpanel/Mysql/Basic.pm line 391, <STDIN> line 1.
[2019-09-16 15:01:41 -0400] warn [cpsrvd] Previous check_sessions still running with pid: 8791.  Possible cpsrvd denial of service attack in progress. at cpsrvd.pl line 1150.
    cpanel::cpsrvd::script() called at cpsrvd.pl line 427
cpsrvd 1693d8f3ff03a: Died at /usr/local/cpanel/Cpanel/Server/Dovecot.pm line 71, <GEN28> line 2.

[2019-09-16 15:04:03 -0400] warn [tailwatchd] [9905] TIME 1568660637 UID 0 clobbered stale lock file “/var/run/chkservd.checking.lock” (NAME “tailwatchd - chkservd”, UID 0) written by PID 8532 at 1568660026 at /usr/local/cpanel/Cpanel/SafeFile.pm line 931.
    Cpanel::SafeFile::_lock_wait("/var/run/chkservd.checking", Cpanel::SafeFileLock=ARRAY(0x19825a0), "/var/run/chkservd.checking.lock") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 350
    Cpanel::SafeFile::_safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 294
    Cpanel::SafeFile::safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 252
    Cpanel::TailWatch::ChkServd::_run_checks(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568660542) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 208
    Cpanel::TailWatch::ChkServd::run(Cpanel::TailWatch::ChkServd=HASH(0x14e45e0), Cpanel::TailWatch=HASH(0x9897a0), 1568660542) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 754
    Cpanel::TailWatch::tail_logs(Cpanel::TailWatch=HASH(0x9897a0)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 657
    eval {...} called at /usr/local/cpanel/Cpanel/TailWatch.pm line 625
    Cpanel::TailWatch::run(Cpanel::TailWatch=HASH(0x9897a0), 0, "resume", 1) called at libexec/tailwatch/tailwatchd.pl line 62
    main::run("--resume") called at libexec/tailwatch/tailwatchd.pl line 35
cpsrvd 105557d8af4db: PID 9934: The cPanel Server operation timed out at cpsrvd.pl line 630.
    ...propagated at /usr/local/cpanel/Cpanel/SafeFile.pm line 218.

[2019-09-16 15:04:13 -0400] info [cpsrvd] Request Timeout: "-" 408 Timeout while creating a secure connection
[2019-09-16 15:04:13 -0400] info [whostmgrd] Internal Server Error: "GET / HTTP/1.1" 500 Error ID 105557d8af4db
    (in cleanup)     (in cleanup)  at /usr/local/cpanel/Cpanel/Epoll.pm line 231 during global destruction.
cpsrvd 105557d8af4db: PID 9948: The cPanel Server operation timed out at cpsrvd.pl line 630.

cpsrvd a3d70b55d9a: PID 9961: The cPanel Server operation timed out at cpsrvd.pl line 630.

[2019-09-16 15:04:55 -0400] info [whostmgrd] Internal Server Error: "GET /favicon.ico HTTP/1.1" 500 Error ID 105557d8af4db
[2019-09-16 15:04:55 -0400] info [whostmgrd] Internal Server Error: "GET / HTTP/1.1" 500 Error ID a3d70b55d9a
cpsrvd a3d70b55d9a: PID 10103: The cPanel Server operation timed out at cpsrvd.pl line 630.

[2019-09-16 15:05:42 -0400] info [whostmgrd] Internal Server Error: "GET /favicon.ico HTTP/1.1" 500 Error ID a3d70b55d9a
cpsrvd a3d70b55d9a: PID 10149: The cPanel Server operation timed out at cpsrvd.pl line 630.

[2019-09-16 15:05:47 -0400] info [whostmgrd] Internal Server Error: "GET / HTTP/1.1" 500 Error ID a3d70b55d9a
cpsrvd a3d70b55d9a: PID 10260: The cPanel Server operation timed out at cpsrvd.pl line 630.

[2019-09-16 15:06:25 -0400] info [whostmgrd] Internal Server Error: "GET /favicon.ico HTTP/1.1" 500 Error ID a3d70b55d9a
[2019-09-16 15:06:26 -0400] warn [pureauth] Failed to call hulk pre_login for [email protected] (system) at bin/pureauth.pl line 136.
cpsrvd a3d70b55d9a: PID 10359: The cPanel Server operation timed out at cpsrvd.pl line 630.

[2019-09-16 15:06:53 -0400] info [whostmgrd] Internal Server Error: "GET / HTTP/1.1" 500 Error ID a3d70b55d9a
cpsrvd a3d70b55d9a: PID 10417: The cPanel Server operation timed out at cpsrvd.pl line 630.
 
Last edited by a moderator:

Jcats

Well-Known Member
PartnerNOC
May 25, 2011
807
160
168
New Jersey
cPanel Access Level
DataCenter Provider
Out of memory

Unfortunately one log file will not sum it up for you in terms of what caused your system to OOM but you could check /var/log/messges and dmesg if you see tons of OOM / killed processes that are PHP related than most likely some kind of layer 7 attack, common scenario would be brute force attack on WordPress.
 
  • Like
Reactions: Topotyn

Topotyn

Member
Sep 19, 2019
6
1
1
Ukraine
cPanel Access Level
Root Administrator
Thanks for the answer! I can provide more logs. I looked through all the Apache logs, nothing unusual. Below added additional logs. If you have time - look, maybe you will find something. Nothing was written to the dmesg that day.

- Removed -
 

Attachments

Last edited:

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,293
1,279
313
Houston
As indicated by @Jcats you are certainly running out of memory. Errors like this provide evidence as such as well:
Code:
    Cpanel::iContact::Class::new("Cpanel::iContact::Class::chkservd::OOM", "process_killed", 1, "seconds_since_boot", 1599891.117932, "uid", 0, "pid", ...) called at /usr/local/cpanel/Cpanel/Notify.pm line 77
I see similar OOM related messages in the output of messages you provided as well. How much memory is on the system? You can find this out by running:
Code:
free -m
You can also get some historical usage over the last 12 hours by running something like:

Code:
sar -r