500 errors during basic tweaks in WHM

LukeFive Dev · Oct 16, 2018

Trying to tweak basic configuration on our VPS, keep getting 500 errors, and now the following...

Code:

(XID j8669k) The system failed to fork a new process because of an error: Cannot allocate memory at /usr/local/cpanel/Cpanel/ForkAsync.pm line 62.
    Cpanel::ForkAsync::do_in_child(CODE(0x4ed1658)) called at /usr/local/cpanel/Cpanel/SafeRun/Object.pm line 363
    Cpanel::SafeRun::Object::new("Cpanel::SafeRun::Object", "program", "/usr/bin/repoquery", "args", ARRAY(0x51c55a8), "before_exec", CODE(0x49e0180)) called at /usr/local/cpanel/Cpanel/SafeRun/Object.pm line 497
    Cpanel::SafeRun::Object::new_or_die("Cpanel::SafeRun::Object", "program", "/usr/bin/repoquery", "args", ARRAY(0x51c55a8), "before_exec", CODE(0x49e0180)) called at /usr/local/cpanel/Cpanel/SysPkgs/RepoQuery.pm line 93
    Cpanel::SysPkgs::RepoQuery::get_all_packages("http://httpupdate.cpanel.net/cpanelsync/repos/CentOS/6/cpanel"..., "name", "summary", "description", "version", "release", "url") called at /usr/local/cpanel/Whostmgr/Addons/RPM/Cache.pm line 162
    Whostmgr::Addons::RPM::Cache::_yum_modules() called at /usr/local/cpanel/Whostmgr/Addons/RPM/Cache.pm line 100
    Whostmgr::Addons::RPM::Cache::_LOAD_FRESH("Whostmgr::Addons::RPM::Cache") called at /usr/local/cpanel/Cpanel/CacheFile.pm line 225
    Cpanel::CacheFile::load("Whostmgr::Addons::RPM::Cache") called at /usr/local/cpanel/Whostmgr/Addons/RPM/Cache.pm line 88
    Whostmgr::Addons::RPM::Cache::load("Whostmgr::Addons::RPM::Cache") called at /usr/local/cpanel/Whostmgr/Addons/RPM.pm line 62
    Whostmgr::Addons::RPM::get_modules() called at whostmgr/bin/whostmgr2.pl line 2362
    main::manage_plugins("manage_plugins") called at /usr/local/cpanel/Whostmgr/Dispatch.pm line 230
    Whostmgr::Dispatch::_do_call("manage_plugins", HASH(0x449d180), HASH(0x44a1120)) called at /usr/local/cpanel/Whostmgr/Dispatch.pm line 144
    Whostmgr::Dispatch::dispatch("manage_plugins", 1, ARRAY(0x44a0d18), HASH(0x44a1120)) called at whostmgr/bin/whostmgr2.pl line 687

SS-Maddy · Oct 16, 2018

Hello @LukeFive Dev

How much RAM does your VPS have ? It seems that you are running out of memory. What is the cPanel memory limit configured in "WHM >> Tweak Settings" . Try upgrading the memory of your VPS by 1GB or so and then see whether the issue persists.

LukeFive Dev · Oct 17, 2018

Hello @SS-Maddy :

I made these two changes and now waiting to see what happens ...

"Max cPanel memory" from 768MB to 1,768MB
"Conserve memory" now OFF (before it was ON)

I'm a relative newbie to back-end work. Please let me know if there is a good tutorial on best practices for WHM/cPanel.

LukeFive Dev · Oct 17, 2018

While I am in the WHM, I notice something unsettling about the status of my services ...

SERVICE STATUS
clamd Down
dnsadmin Down
spamd Down

LukeFive Dev · Oct 17, 2018

@SS-Maddy I am not sure how often this thread may be checked, so an update.
I am now rebuilding the RPM database. Shortly before that I retrieved the error log. Here are the last 100 lines (scrubbed) ...

Code:

open3: fork failed: Cannot allocate memory at /usr/local/cpanel/Cpanel/Services.pm line 173.
[2018-10-17 08:29:06 -0700] warn [whostmgrd] The subprocess (whostmgr (whostmgr)) exited with an error: The subprocess reported error number 2 when it ended. at /usr/local/cpanel/Cpanel/Server/Handlers/SubProcess.pm line 251.
        Cpanel::Server::Handlers::SubProcess::_report_subprocess_errors(Cpanel::Server::Handlers::SubProcess=HASH(0x2a8d8c8)) called at /usr/local/cpanel/Cpanel/Server/Handlers/SubProcess.pm line 109
        Cpanel::Server::Handlers::SubProcess::handler(Cpanel::Server::Handlers::SubProcess=HASH(0x2a8d8c8), "subprocess_name", "whostmgr (whostmgr)", "subprocess_read_handle", IO::Handle=GLOB(0x2a0c840), "subprocess_write_handle", IO::Handle=GLOB(0x2a8d568), "subprocess_pid_to_reap", ...) called at cpsrvd.pl line 6819
        cpanel::cpsrvd::cpHandler("app", "whostmgr", "json", 0, "document", "./scripts/resmysql") called at cpsrvd.pl line 5979
        cpanel::cpsrvd::dodoc_whostmgrd() called at cpsrvd.pl line 1683
        cpanel::cpsrvd::dodoc(HASH(0x1347a00)) called at cpsrvd.pl line 1456
        cpanel::cpsrvd::handle_one_connection(7) called at cpsrvd.pl line 894
        cpanel::cpsrvd::script() called at cpsrvd.pl line 325
[2018-10-17 08:29:07 -0700] warn [cpsrvd] Failed to fork(): for main connection loop: Cannot allocate memory at cpsrvd.pl line 898.
        cpanel::cpsrvd::script() called at cpsrvd.pl line 325
[2018-10-17 08:29:07 -0700] warn [cpsrvd] Failed to fork(): for main connection loop: Cannot allocate memory at cpsrvd.pl line 898.
        cpanel::cpsrvd::script() called at cpsrvd.pl line 325
[2018-10-17 08:29:08 -0700] warn [cpsrvd] Failed to fork(): for main connection loop: Cannot allocate memory at cpsrvd.pl line 898.
        cpanel::cpsrvd::script() called at cpsrvd.pl line 325
[2018-10-17 08:29:08 -0700] warn [cpsrvd] Failed to fork(): for main connection loop: Cannot allocate memory at cpsrvd.pl line 898.
        cpanel::cpsrvd::script() called at cpsrvd.pl line 325
[2018-10-17 08:29:08 -0700] warn [cpsrvd] Failed to fork(): for main connection loop: Cannot allocate memory at cpsrvd.pl line 898.
        cpanel::cpsrvd::script() called at cpsrvd.pl line 325
[2018-10-17 08:29:08 -0700] warn [cpsrvd] Failed to fork(): for main connection loop: Cannot allocate memory at cpsrvd.pl line 898.
        cpanel::cpsrvd::script() called at cpsrvd.pl line 325
[2018-10-17 08:29:08 -0700] warn [cpsrvd] Failed to fork(): for main connection loop: Cannot allocate memory at cpsrvd.pl line 898.
        cpanel::cpsrvd::script() called at cpsrvd.pl line 325
Out of memory!
Out of memory!
[2018-10-17 08:29:09 -0700] warn [cpsrvd] Failed to fork(): for main connection loop: Cannot allocate memory at cpsrvd.pl line 898.
        cpanel::cpsrvd::script() called at cpsrvd.pl line 325
[2018-10-17 08:32:49 -0700] warn [tailwatchd] [21788] TIME 1539790369 UID 0 clobbered stale lock file “/var/run/chkservd.checking.lock” (NAME“tailwatchd - chkservd”, UID 0) written by PID 19903 at 1539790068 at /usr/local/cpanel/Cpanel/SafeFile.pm line 913.
        Cpanel::SafeFile::_lock_wait("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 355
        Cpanel::SafeFile::_safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 293
        Cpanel::SafeFile::safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 252
        Cpanel::TailWatch::ChkServd::_run_checks(Cpanel::TailWatch::ChkServd=HASH(0x1bee698), Cpanel::TailWatch=HASH(0x7c3c68), 1539790369) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 208
        Cpanel::TailWatch::ChkServd::run(Cpanel::TailWatch::ChkServd=HASH(0x1bee698), Cpanel::TailWatch=HASH(0x7c3c68), 1539790369) called at/usr/local/cpanel/Cpanel/TailWatch.pm line 748
        Cpanel::TailWatch::tail_logs(Cpanel::TailWatch=HASH(0x7c3c68)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 648
        eval {...} called at /usr/local/cpanel/Cpanel/TailWatch.pm line 616
        Cpanel::TailWatch::run(Cpanel::TailWatch=HASH(0x7c3c68)) called at libexec/tailwatch/tailwatchd.pl line 58
        main::run("--start") called at libexec/tailwatch/tailwatchd.pl line 35
[2018-10-17 08:32:49 -0700] info [tailwatchd] chkservd::Hang Notification => info@s000-000-000-000.secureserve***** via EMAIL [eventimportance => High (1)]
==> cpsrvd 11.74.0.9 started
==> cpsrvd: loading security policy....Done
==> cpsrvd: Setting up SSL support ... Done
==> cpsrvd: transferred port bindings
==> cpsrvd: bound to ports
[2018-10-17 15:33:12 +0000] warn [restartsrv_spamd] The 'spamd' service's PID file '/var/run/spamd.pid' did not appear after 10 seconds.
[2018-10-17 08:33:16 -0700] info [queueprocd] chkservd::Notify Notification => info@s000-000-000-000.secureserve***** via EMAIL [eventimportance => High (1)]
==> cpsrvd 11.74.0.9 started
==> cpsrvd: loading security policy....Done
==> cpsrvd: Setting up SSL support ... Done
==> cpsrvd: transferred port bindings
==> cpsrvd: bound to ports
[2018-10-17 08:40:02 -0700] info [queueprocd] chkservd::Notify Notification => info@s000-000-000-000.secureserve***** via EMAIL [eventimportance => High (1)]
[2018-10-17 15:41:00 +0000] warn [restartsrv_clamd] The system encountered an error while starting the “clamd” service with the command “/usr/local/cpanel/3rdparty/bin/clamd”: The subprocess reported error number 1 when it ended. at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 1403.
        Cpanel::ServiceManager::Base::warn(Cpanel::ServiceManager::Services::Clamd=HASH(0x10dd6f8), "The system encountered an error while starting the \x{e2}\x{80}\x{9c}clamd\x{e2}\x{80}"...) called at /usr/local/cpanel/Cpanel/ServiceManager/Manager/Initd.pm line 103
        Cpanel::ServiceManager::Manager::Initd::start(Cpanel::ServiceManager::Manager::Initd=HASH(0x113dbd0), Cpanel::ServiceManager::Services::Clamd=HASH(0x10dd6f8)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 848
        Cpanel::ServiceManager::Base::start(Cpanel::ServiceManager::Services::Clamd=HASH(0x10dd6f8)) called at /usr/local/cpanel/Cpanel/ServiceManager/Services/Clamd.pm line 60
        Cpanel::ServiceManager::Services::Clamd::restart(Cpanel::ServiceManager::Services::Clamd=HASH(0x10dd6f8), "graceful", undef) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 540
        Cpanel::ServiceManager::Base::run_from_argv(Cpanel::ServiceManager::Services::Clamd=HASH(0x10dd6f8), "--verbose", "--restart", "--attempt", 99) called at bin/restartsrv_base.pl line 96
        scripts::restartsrv_base::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90
        Try::Tiny::try(CODE(0x10e1cb8), Try::Tiny::Catch=REF(0xb44f68)) called at bin/restartsrv_base.pl line 130
        scripts::restartsrv_base::run("--restart", "--attempt", 99) called at bin/restartsrv_base.pl line 33
[2018-10-17 08:41:02 -0700] info [queueprocd] chkservd::Notify Notification => info@s000-000-000-000.secureserve***** via EMAIL [eventimportance => High (1)]
[2018-10-17 08:41:02 -0700] info [queueprocd] chkservd::Notify Notification => info@s000-000-000-000.secureserve***** via EMAIL [eventimportance => High (1)]
[2018-10-17 08:41:54 -0700] info [xml-api] Syncing version information from httpupdate.cpanel.net/cpanelsync/TIERS.json
[2018-10-17 08:41:56 -0700] info [xml-api] Successfully verified signature for cpanel (key types: release).
[2018-10-17 08:41:58 -0700] info [whostmgrd] Internal Server Error: "GET /cpsess6115035217/scripts/reshttpd?confirm=1 HTTP/1.1" 500 No response from subprocess (whostmgr (whostmgr)):
[2018-10-17 08:42:05 -0700] warn [cpsrvd] Failed to fork(): for main connection loop: Cannot allocate memory at cpsrvd.pl line 898.
        cpanel::cpsrvd::script() called at cpsrvd.pl line 325
[2018-10-17 08:42:34 -0700] info [whostmgrd] Internal Server Error: "GET /cpsess6115035217/scripts/servup HTTP/1.1" 500 No response from subprocess (whostmgr (whostmgr)):
[2018-10-17 15:43:14 +0000] warn [restartsrv_spamd] The 'spamd' service's PID file '/var/run/spamd.pid' did not appear after 10 seconds.
[2018-10-17 08:43:18 -0700] info [queueprocd] chkservd::Notify Notification => info@s000-000-000-000.secureserve***** via EMAIL [eventimportance => High (1)]
[2018-10-17 08:47:52 -0700] warn [tailwatchd] [26365] TIME 1539791272 UID 0 clobbered stale lock file “/var/run/chkservd.checking.lock” (NAME“tailwatchd - chkservd”, UID 0) written by PID 25971 at 1539790971 at /usr/local/cpanel/Cpanel/SafeFile.pm line 913.
        Cpanel::SafeFile::_lock_wait("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 355
        Cpanel::SafeFile::_safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 293
        Cpanel::SafeFile::safelock("/var/run/chkservd.checking") called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 252
        Cpanel::TailWatch::ChkServd::_run_checks(Cpanel::TailWatch::ChkServd=HASH(0x1bee698), Cpanel::TailWatch=HASH(0x7c3c68), 1539791272) called at /usr/local/cpanel/Cpanel/TailWatch/ChkServd.pm line 208
        Cpanel::TailWatch::ChkServd::run(Cpanel::TailWatch::ChkServd=HASH(0x1bee698), Cpanel::TailWatch=HASH(0x7c3c68), 1539791272) called at/usr/local/cpanel/Cpanel/TailWatch.pm line 748
        Cpanel::TailWatch::tail_logs(Cpanel::TailWatch=HASH(0x7c3c68)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 648
        eval {...} called at /usr/local/cpanel/Cpanel/TailWatch.pm line 616
        Cpanel::TailWatch::run(Cpanel::TailWatch=HASH(0x7c3c68)) called at libexec/tailwatch/tailwatchd.pl line 58
        main::run("--start") called at libexec/tailwatch/tailwatchd.pl line 35
[2018-10-17 08:47:52 -0700] info [tailwatchd] chkservd::Hang Notification => info@s000-000-000-000.secureserve***** via EMAIL [eventimportance => High (1)]
[2018-10-17 08:48:05 -0700] info [queueprocd] chkservd::Notify Notification => info@s000-000-000-000.secureserve***** via EMAIL [eventimportance => High (1)]
[2018-10-17 08:54:51 -0700] info [queueprocd] chkservd::Notify Notification => info@s000-000-000-000.secureserve***** via EMAIL [eventimportance => High (1)]
[2018-10-17 15:55:50 +0000] warn [restartsrv_clamd] The system encountered an error while starting the “clamd” service with the command “/usr/local/cpanel/3rdparty/bin/clamd”: The subprocess reported error number 1 when it ended. at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 1403.
        Cpanel::ServiceManager::Base::warn(Cpanel::ServiceManager::Services::Clamd=HASH(0x29506f8), "The system encountered an error while starting the \x{e2}\x{80}\x{9c}clamd\x{e2}\x{80}"...) called at /usr/local/cpanel/Cpanel/ServiceManager/Manager/Initd.pm line 103
        Cpanel::ServiceManager::Manager::Initd::start(Cpanel::ServiceManager::Manager::Initd=HASH(0x29b0ba0), Cpanel::ServiceManager::Services::Clamd=HASH(0x29506f8)) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 848
        Cpanel::ServiceManager::Base::start(Cpanel::ServiceManager::Services::Clamd=HASH(0x29506f8)) called at /usr/local/cpanel/Cpanel/ServiceManager/Services/Clamd.pm line 60
        Cpanel::ServiceManager::Services::Clamd::restart(Cpanel::ServiceManager::Services::Clamd=HASH(0x29506f8), "graceful", undef) called at /usr/local/cpanel/Cpanel/ServiceManager/Base.pm line 540
        Cpanel::ServiceManager::Base::run_from_argv(Cpanel::ServiceManager::Services::Clamd=HASH(0x29506f8), "--verbose", "--restart", "--attempt", 100) called at bin/restartsrv_base.pl line 96
        scripts::restartsrv_base::__ANON__() called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 99
        eval {...} called at /usr/local/cpanel/3rdparty/perl/526/lib64/perl5/cpanel_lib/Try/Tiny.pm line 90
        Try::Tiny::try(CODE(0x2954c88), Try::Tiny::Catch=REF(0xb44f68)) called at bin/restartsrv_base.pl line 130
        scripts::restartsrv_base::run("--restart", "--attempt", 100) called at bin/restartsrv_base.pl line 33
[2018-10-17 08:55:54 -0700] info [queueprocd] chkservd::Notify Notification => info@s000-000-000-000.secureserve***** via EMAIL [eventimportance => High (1)]
[2018-10-17 08:58:02 -0700] info [queueprocd] chkservd::Notify Notification => info@s000-000-000-000.secureserve***** via EMAIL [eventimportance => High (1)]

cPanelLauren · Oct 17, 2018

Hi @LukeFive Dev

I think at this point it might be important to know how much memory the server has allocated to it. You can see this at WHM>>Server Status>>Server Information.

Thanks!

LukeFive Dev · Oct 18, 2018

@cPanelLauren here is both Current Memory Usage and Current Disk Usage ...
total used free shared buffers cached
Mem: 2097152 1664052 433100 267816 0 601528
-/+ buffers/cache: 1062524 1034628
Swap: 0 0 0
Total: 2097152 1664052 433100

Filesystem Size Used Avail Use% Mounted on
/dev/ploop54140p1 59G 20G 37G 36% /
none 1.0G 4.0K 1.0G 1% /dev
none 1.0G 0 1.0G 0% /dev/shm

cPanelLauren · Oct 19, 2018

Hi @LukeFive Dev

That first one is what I wanted but it shows you have plenty of memory available. When you're experiencing the issue I'm curious what it reports.

Could you run the following (just to get a bit of historical data for the day)
Code:
sar -r 
Also, do you have shell fork bomb protection enabled on the server? You can find this at WHM>>Security Center>>Shell Fork Bomb Protection. If you do, could you please disable it and let me know if the issue persists.

Thanks!

LukeFive Dev · Oct 19, 2018

@cPanelLauren Great difficulty trying to login today. Had to power off/on. Shell Fork Bomb Protection disabled. Here is the sar :

Code:

 sar -r
Linux 2.6.32-042stab120.16 (s000-000-000-000.secureserver.net)  10/19/2018      _x86_64_        (120 CPU)
12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
12:10:02 AM         0   2097152    100.00         0   1118460         0      0.00
12:20:01 AM         0   2097152    100.00         0   1114816         0      0.00
12:40:01 AM         0   2097152    100.00         0   1112640         0      0.00
12:50:01 AM         0   2097152    100.00         0   1112996         0      0.00
01:10:01 AM         0   2097152    100.00         0   1111620         0      0.00
01:20:01 AM         0   2097152    100.00         0   1108932         0      0.00
01:40:01 AM         0   2097152    100.00         0   1096824         0      0.00
01:50:02 AM         0   2097152    100.00         0   1096820         0      0.00
02:00:01 AM         0   2097152    100.00         0   1096988         0      0.00
02:10:01 AM         0   2097152    100.00         0   1099464         0      0.00
02:20:01 AM         0   2097152    100.00         0   1099492         0      0.00
02:30:01 AM         0   2097152    100.00         0   1099736         0      0.00
02:40:01 AM         0   2097152    100.00         0   1100428         0      0.00
02:50:01 AM         0   2097152    100.00         0   1083336         0      0.00
03:00:01 AM         0   2097152    100.00         0   1065172         0      0.00
03:10:01 AM         0   2097152    100.00         0   1065300         0      0.00
03:20:01 AM         0   2097152    100.00         0   1065268         0      0.00
03:30:02 AM         0   2097152    100.00         0   1064708         0      0.00
03:40:01 AM         0   2097152    100.00         0   1060732         0      0.00
03:50:01 AM         0   2097152    100.00         0   1065940         0      0.00
04:00:01 AM         0   2097152    100.00         0   1065948         0      0.00
04:10:01 AM         0   2097152    100.00         0   1066484         0      0.00
04:10:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
04:20:01 AM         0   2097152    100.00         0    734800         0      0.00
04:30:01 AM         0   2097152    100.00         0    757216         0      0.00
04:40:01 AM         0   2097152    100.00         0    759624         0      0.00
04:50:01 AM         0   2097152    100.00         0    757556         0      0.00
05:00:01 AM         0   2097152    100.00         0    758316         0      0.00
05:10:01 AM         0   2097152    100.00         0    757796         0      0.00
05:20:01 AM         0   2097152    100.00         0    760668         0      0.00
05:30:01 AM         0   2097152    100.00         0    765628         0      0.00
05:40:01 AM         0   2097152    100.00         0    774808         0      0.00
05:50:01 AM         0   2097152    100.00         0    774380         0      0.00
06:00:01 AM         0   2097152    100.00         0    773540         0      0.00
06:10:01 AM         0   2097152    100.00         0    768840         0      0.00
06:20:01 AM         0   2097152    100.00         0    754728         0      0.00
06:30:01 AM         0   2097152    100.00         0    754980         0      0.00
06:40:01 AM         0   2097152    100.00         0    758388         0      0.00
06:50:01 AM         0   2097152    100.00         0    758456         0      0.00
07:00:01 AM         0   2097152    100.00         0    758764         0      0.00
07:10:02 AM         0   2097152    100.00         0    758868         0      0.00
07:20:01 AM         0   2097152    100.00         0    753324         0      0.00
07:30:01 AM    147012   1950140     92.99         0    777664         0      0.00
07:40:01 AM    147860   1949292     92.95         0    777892         0      0.00
07:50:01 AM    151076   1946076     92.80         0    778448         0      0.00
07:50:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
08:00:01 AM         0   2097152    100.00         0   1191780         0      0.00
08:10:01 AM         0   2097152    100.00         0   1185560         0      0.00
08:20:01 AM         0   2097152    100.00         0   1178756         0      0.00
08:30:01 AM         0   2097152    100.00         0   1180992         0      0.00
08:40:01 AM         0   2097152    100.00         0   1176740         0      0.00
08:50:01 AM         0   2097152    100.00         0   1178204         0      0.00
09:00:01 AM         0   2097152    100.00         0   1182032         0      0.00
Average:         8744   2088408     99.58         0    961193         0      0.00
09:08:44 AM       LINUX RESTART

LukeFive Dev · Oct 19, 2018

Trying to be proactive here is result of top ...

Code:

top - 09:34:48 up 26 min,  0 users,  load average: 0.13, 0.10, 0.08
Tasks:  54 total,   1 running,  52 sleeping,   0 stopped,   1 zombie
Cpu(s):  0.2%us,  0.2%sy,  0.5%ni, 99.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2097152k total,  1717032k used,   380120k free,        0k buffers
Swap:        0k total,        0k used,        0k free,   658316k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4261 admin     20   0  546m  30m  18m S  4.7  1.5   0:00.14 php-fpm: pool DOMAIN_COM
 1333 mysql     20   0 1420m  50m 8404 S  0.7  2.5   0:03.24 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/
 2182 nobody    20   0 2574m  18m 8452 S  0.3  0.9   0:00.50 /usr/sbin/httpd
 2278 cpanelco  20   0 12872 4800 4348 S  0.3  0.2   0:02.37 /usr/local/cpanel/3rdparty/sbin/p0f -i any -u cpanelconnecttrack -d -s /var/cpane
 4181 root      20   0 15000 1284 1008 R  0.3  0.1   0:00.06 top c
    1 root      20   0 19240 1540 1272 S  0.0  0.1   0:00.04 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 [kthreadd/176124]
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.00 [khelper/176124]
  134 root      16  -4 10648  596  324 S  0.0  0.0   0:00.00 /sbin/udevd -d
  702 root      20   0  179m 1476 1064 S  0.0  0.1   0:00.03 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
  724 named     20   0 1616m  32m 2600 S  0.0  1.6   0:00.07 /usr/sbin/named -u named
  764 dbus      20   0 21444  888  636 S  0.0  0.0   0:00.00 dbus-daemon --system
 1029 cpanelso  20   0 5133m 245m  20m S  0.0 12.0   0:14.78 /usr/lib/jvm/jre-1.8.0/bin/java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:Surv
 1164 root      20   0 66296 1236  492 S  0.0  0.1   0:00.00 /usr/sbin/sshd
 1175 root      20   0 21720  952  716 S  0.0  0.0   0:00.00 xinetd -stayalive -pidfile /var/run/xinetd.pid
 1185 root      20   0  105m 1644 1332 S  0.0  0.1   0:00.02 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/s
 1371 root      20   0 19984 1172  832 S  0.0  0.1   0:00.01 /usr/sbin/dovecot

cPanelLauren · Oct 19, 2018

HI @LukeFive Dev

That's some really interesting usage output there. While your commit% is at 0 your usage is 100% across the board. That's not normal behavior and would definitely explain why you're having trouble accessing the server. None the less 2097152k translates to 2GB of memory which is really the bare minimum.

If your provider offers support I would suggest contacting them to address the high memory usage, I don't know if you're on a VM or a hardware server but if you're on a VM it may be worth investigating the IO usage as well. You might also check out the tutorial here: Tutorial - Troubleshooting high server loads on Linux servers (while your load average is low your memory usage is quite high at least according to sar and for the amount of memory you have on the server)

Thanks!

LukeFive Dev · Oct 19, 2018

Well it's a 2GB VPS on GoDaddy, and the account basically has no support. I will try the tutorial, but after that try opening a ticket. The two larger sites we have there are noticing the error messages and wonder why our usually speedy service, this time is not.
Thank you Lauren and others for analysis of this item.

cPanelLauren · Oct 19, 2018

Hi @LukeFive Dev

Can you run this:
Code:
sar -p 
It'll give you an idea of the IO wait. Seemingly phantom IO wait and high memory usage (sometimes along with high load) is not uncommon on VPS's on overloaded host nodes.

LukeFive Dev · Oct 19, 2018

@cPanelLauren - the restarts are me trying to get it stable enough to log in...
10/19/2018 _x86_64_ (120 CPU)

Code:

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:02 AM     all      0.03      0.00      0.02      0.00      0.00     99.96
12:20:01 AM     all      0.03      0.00      0.01      0.00      0.00     99.95
12:40:01 AM     all      0.03      0.00      0.01      0.01      0.00     99.94
12:50:01 AM     all      0.03      0.00      0.02      0.00      0.00     99.95
01:10:01 AM     all      0.03      0.00      0.02      0.01      0.00     99.94
01:20:01 AM     all      0.02      0.00      0.01      0.00      0.00     99.97
01:40:01 AM     all      0.02      0.00      0.02      0.00      0.00     99.96
01:50:02 AM     all      0.02      0.00      0.01      0.00      0.00     99.97
02:00:01 AM     all      0.02      0.00      0.01      0.00      0.00     99.97
02:10:01 AM     all      0.03      0.00      0.02      0.00      0.00     99.95
02:20:01 AM     all      0.04      0.00      0.01      0.01      0.00     99.94
02:30:01 AM     all      0.03      0.00      0.01      0.00      0.00     99.95
02:40:01 AM     all      0.04      0.00      0.02      0.00      0.00     99.94
02:50:01 AM     all      0.02      0.00      0.02      0.03      0.00     99.93
03:00:01 AM     all      0.03      0.00      0.01      0.00      0.00     99.95
03:10:01 AM     all      0.03      0.00      0.01      0.00      0.00     99.95
03:20:01 AM     all      0.02      0.00      0.01      0.00      0.00     99.96
03:30:02 AM     all      0.02      0.00      0.01      0.00      0.00     99.96
03:40:01 AM     all      0.05      0.00      0.02      0.01      0.00     99.92
03:50:01 AM     all      0.04      0.00      0.02      0.00      0.00     99.94
04:00:01 AM     all      0.02      0.00      0.01      0.00      0.00     99.97
04:10:01 AM     all      0.04      0.01      0.02      0.00      0.00     99.94
04:10:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:20:01 AM     all      0.03      0.00      0.02      0.00      0.00     99.95
04:30:01 AM     all      0.08      0.00      0.04      0.01      0.00     99.88
04:40:01 AM     all      0.02      0.00      0.01      0.00      0.00     99.96
04:50:01 AM     all      0.02      0.00      0.01      0.01      0.00     99.96
05:00:01 AM     all      0.02      0.00      0.01      0.00      0.00     99.96
05:10:01 AM     all      0.03      0.00      0.01      0.00      0.00     99.96
05:20:01 AM     all      0.03      0.00      0.02      0.00      0.00     99.95
05:30:01 AM     all      0.03      0.00      0.01      0.00      0.00     99.95
05:40:01 AM     all      0.05      0.00      0.02      0.06      0.00     99.86
05:50:01 AM     all      0.02      0.00      0.01      0.00      0.00     99.96
06:00:01 AM     all      0.02      0.00      0.02      0.01      0.00     99.95
06:10:01 AM     all      0.03      0.00      0.02      0.00      0.00     99.95
06:20:01 AM     all      0.04      0.00      0.02      0.00      0.00     99.93
06:30:01 AM     all      0.03      0.00      0.01      0.00      0.00     99.96
06:40:01 AM     all      0.04      0.00      0.02      0.00      0.00     99.93
06:50:01 AM     all      0.02      0.00      0.01      0.01      0.00     99.96
07:00:01 AM     all      0.04      0.00      0.01      0.00      0.00     99.95
07:10:02 AM     all      0.03      0.00      0.02      0.02      0.00     99.93
07:20:01 AM     all      0.02      0.00      0.01      0.00      0.00     99.97
07:30:01 AM     all      0.02      0.03      0.01      0.00      0.00     99.94
07:40:01 AM     all      0.02      0.00      0.01      0.00      0.00     99.96
07:50:01 AM     all      0.03      0.00      0.02      0.00      0.00     99.95
07:50:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
08:00:01 AM     all      0.84      0.00      0.47      0.85      0.00     97.84
08:10:01 AM     all      0.04      0.00      0.02      0.01      0.00     99.93
08:20:01 AM     all      0.03      0.01      0.02      0.01      0.00     99.94
08:30:01 AM     all      0.02      0.00      0.01      0.00      0.00     99.96
08:40:01 AM     all      0.05      0.00      0.02      0.02      0.00     99.91
08:50:01 AM     all      0.04      0.00      0.02      0.00      0.00     99.93
09:00:01 AM     all      0.04      0.00      0.02      0.01      0.00     99.93
Average:        all      0.04      0.00      0.02      0.02      0.00     99.91
09:08:44 AM       LINUX RESTART
09:10:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
09:20:01 AM     all      0.09      0.00      0.04      0.03      0.00     99.85
09:30:01 AM     all      0.06      0.00      0.02      0.17      0.00     99.74
09:40:01 AM     all      0.16      0.00      0.04      0.02      0.00     99.78
09:50:01 AM     all      0.12      0.00      0.03      0.17      0.00     99.68
10:00:01 AM     all      0.08      0.00      0.03      0.00      0.00     99.89
10:10:02 AM     all      0.16      0.00      0.04      0.07      0.00     99.73
10:20:01 AM     all      0.16      0.00      0.04      0.04      0.00     99.76
10:30:01 AM     all      0.12      0.00      0.03      0.03      0.00     99.82
10:40:01 AM     all      0.11      0.00      0.03      0.01      0.00     99.85
10:50:01 AM     all      0.13      0.01      0.04      0.00      0.00     99.83
11:00:01 AM     all      0.09      0.00      0.03      0.06      0.00     99.82
11:10:01 AM     all      0.09      0.00      0.03      0.24      0.00     99.64
11:20:01 AM     all      0.09      0.00      0.03      0.00      0.00     99.87
11:30:01 AM     all      0.11      0.00      0.04      0.03      0.00     99.82
11:40:01 AM     all      0.05      0.00      0.02      0.00      0.00     99.93
11:50:01 AM     all      0.07      0.00      0.03      0.01      0.00     99.89
12:00:01 PM     all      0.03      0.00      0.01      0.00      0.00     99.96
12:10:01 PM     all      0.03      0.00      0.02      0.00      0.00     99.95
12:20:01 PM     all      0.20      0.00      0.04      0.14      0.00     99.62
12:30:01 PM     all      0.04      0.00      0.01      0.00      0.00     99.94
12:40:01 PM     all      0.10      0.00      0.03      0.03      0.00     99.85
12:50:01 PM     all      0.11      0.00      0.03      0.00      0.00     99.86
12:50:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
01:00:01 PM     all      0.07      0.00      0.02      0.00      0.00     99.90
01:10:01 PM     all      0.07      0.00      0.02      0.00      0.00     99.91
01:20:01 PM     all      0.05      0.01      0.02      0.00      0.00     99.92
01:30:01 PM     all      0.05      0.00      0.02      0.02      0.00     99.90
01:40:01 PM     all      0.12      0.00      0.04      0.07      0.00     99.78
01:50:01 PM     all      0.61      0.00      0.08      0.05      0.00     99.26
02:00:01 PM     all      0.07      0.00      0.03      0.00      0.00     99.90
02:20:01 PM     all      0.08      0.00      0.03      0.04      0.00     99.85
Average:        all      0.11      0.00      0.03      0.04      0.00     99.82
02:21:44 PM       LINUX RESTART
02:30:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
02:40:01 PM     all      0.16      0.00      0.04      0.04      0.00     99.75
02:50:01 PM     all      0.29      0.00      0.07      0.14      0.00     99.49
03:00:01 PM     all      0.12      0.00      0.04      0.11      0.00     99.73
03:10:01 PM     all      0.10      0.00      0.03      0.25      0.00     99.62
03:20:01 PM     all      0.06      0.00      0.02      0.02      0.00     99.90
03:30:01 PM     all      0.08      0.00      0.03      0.02      0.00     99.87
03:40:01 PM     all      0.05      0.00      0.02      0.00      0.00     99.93
03:50:01 PM     all      0.08      0.00      0.02      0.01      0.00     99.89
Average:        all      0.12      0.00      0.03      0.07      0.00     99.77
03:53:12 PM       LINUX RESTART
04:00:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:10:01 PM     all      0.04      0.00      0.02      0.01      0.00     99.92
04:20:01 PM     all      0.02      0.01      0.01      0.00      0.00     99.96
04:30:01 PM     all      0.07      0.00      0.03      0.04      0.00     99.86
04:40:01 PM     all      0.09      0.00      0.03      0.04      0.00     99.83
04:50:01 PM     all      0.13      0.00      0.04      0.02      0.00     99.81
Average:        all      0.07      0.00      0.03      0.02      0.00     99.88
05:00:05 PM       LINUX RESTART

cPanelLauren · Oct 22, 2018

HI @LukeFive Dev

The IOwait is really quite low and points to something else being the source of the issue you're experiencing. It's possible that you go through the tutorial I provided on troubleshooting high load and find that your traffic warrants more memory/resources.

LukeFive Dev · Oct 22, 2018

@cPanelLauren Yes I am pursuing the tutorial and also opening an official ticket. Thanks for your support!

Forums

The Community Forums

Interact with an entire community of cPanel & WHM users!

500 errors during basic tweaks in WHM

LukeFive Dev Member

SS-Maddy Well-Known Member

LukeFive Dev Member

LukeFive Dev Member

LukeFive Dev Member

cPanelLauren Forums Analyst II Staff Member

LukeFive Dev Member

cPanelLauren Forums Analyst II Staff Member

LukeFive Dev Member

LukeFive Dev Member

cPanelLauren Forums Analyst II Staff Member

LukeFive Dev Member

cPanelLauren Forums Analyst II Staff Member

LukeFive Dev Member

cPanelLauren Forums Analyst II Staff Member

LukeFive Dev Member

SOLVED [CPANEL-23908] "authnlinks" errors during backups/transfers

The system encountered errors during transport error

Mysql errors after upgrading to 5.7

Timeout errors with Cloudflare CDN when running xml php script cron

SOLVED SSL / TLS errors

Share This Page

Useful Searches

The Community Forums

Interact with an entire community of cPanel & WHM users!

500 errors during basic tweaks in WHM

LukeFive Dev Member

SS-Maddy Well-Known Member

LukeFive Dev Member

LukeFive Dev Member

LukeFive Dev Member

cPanelLauren Forums Analyst II Staff Member

LukeFive Dev Member

cPanelLauren Forums Analyst II Staff Member

LukeFive Dev Member

LukeFive Dev Member

cPanelLauren Forums Analyst II Staff Member

LukeFive Dev Member

cPanelLauren Forums Analyst II Staff Member

LukeFive Dev Member

cPanelLauren Forums Analyst II Staff Member

LukeFive Dev Member

SOLVED [CPANEL-23908] "authnlinks" errors during backups/transfers

The system encountered errors during transport error

Mysql errors after upgrading to 5.7

Timeout errors with Cloudflare CDN when running xml php script cron

SOLVED SSL / TLS errors

Share This Page