SOLVED High Load every day at 10:00 am

[mjimeneznet]

Hi, since 22 November, everyday at 10:00 the load of the server starts to increase until I have to reboot because it becomes unusable.

Since then, I'm monitoring everyday with this guide Troubleshooting high server loads on Linux servers and another tools that I use for my work. I've been working with WHM/Cpanel for almost 8 years.

Today, at 10:00 am I stopped mysql, apache, exim, etc... everything. No crontabs between 9:00 am and 11:00 am and the load starts increasing until I lost the server.
All processes are running with 0% cpu

I only see in /var/log/messages is that, everyday at same hour since 22 November.

Dec 2 10:02:14 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:02:18 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:02:31 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:02:42 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:03:04 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:03:33 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:03:55 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:04:27 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:04:49 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:05:18 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:05:42 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:06:15 s107 dbus-daemon: [system] Reloaded configuration
Dec 2 10:06:39 s107 dbus-daemon: [system] Reloaded configuration

Any help is really appreciated.

 

[SysSachin]

Hello,

You have to monitor your server using top command so that you will get exact idea which process causing the high load.

Also, Please check if there any cron set for that time.

 

[mjimeneznet]

As i wrote in the first post, no crons, and all processes stopped. The output of the top when server was about to reach the limit and I rebooted it.

The server is 4 CPU and 16 GB RAM, I rebooted when load was at 13 (1 min avg)

Top (nothing is consuming that CPU)

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  19236  1164 ?        Ss   Dec01   0:00 init
root         2  0.0  0.0      0     0 ?        S    Dec01   0:00 [kthreadd/122537]
root         3  0.0  0.0      0     0 ?        S    Dec01   0:00  \_ [khelper/122537]
root       249  0.0  0.0  10648   260 ?        S<s  Dec01   0:00 /sbin/udevd -d
root       886  0.0  0.0 183556  1804 ?        Sl   Dec01   0:05 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
named      908  0.0  0.0 404324 10684 ?        Ssl  Dec01   0:05 /usr/sbin/named -u named
dbus       928  0.0  0.0  21568  1132 ?        Ss   Dec01   0:00 dbus-daemon --system
root       979  0.0  0.0  66240   548 ?        Ss   Dec01   0:00 /usr/sbin/sshd
root      7143  0.0  0.0 102544  4508 ?        Ss   09:23   0:00  \_ sshd: mjimeneznet [priv]
500       7152  0.0  0.0 102544  2048 ?        S    09:23   0:00  |   \_ sshd: [email protected]/0
500       7154  0.0  0.0 110520  1940 pts/0    Ss   09:23   0:00  |       \_ -bash
root      7184  0.0  0.0 171044  2640 pts/0    S    09:23   0:00  |           \_ sudo -s
root      7186  0.0  0.0 108440  2000 pts/0    S    09:23   0:00  |               \_ /bin/bash
root     27154  0.0  0.0 106072  1472 pts/0    S+   10:05   0:00  |                   \_ bash status.sh
root     27190  0.0  0.0 110524  1336 pts/0    R+   10:05   0:00  |                       \_ ps auxwwwf
root      7436  0.0  0.0 102544  4516 ?        Ss   09:24   0:00  \_ sshd: mjimeneznet [priv]
500       7440  0.0  0.0 102544  2056 ?        S    09:24   0:00  |   \_ sshd: [email protected]/1
500       7442  0.0  0.0 110520  1936 pts/1    Ss   09:24   0:00  |       \_ -bash
root      7465  0.0  0.0 171044  2636 pts/1    S    09:24   0:00  |           \_ sudo -s
root      7467  0.0  0.0 108312  1956 pts/1    S    09:24   0:00  |               \_ /bin/bash
root      7480  0.3  0.0  15144  1512 pts/1    S+   09:24   0:08  |                   \_ top
root      7568  0.0  0.0 102544  4512 ?        Ss   09:25   0:00  \_ sshd: mjimeneznet [priv]
500       7593  0.0  0.0 102544  2060 ?        S    09:25   0:00  |   \_ sshd: [email protected]/3
500       7595  0.0  0.0 110520  1940 pts/3    Ss   09:25   0:00  |       \_ -bash
root      7618  0.0  0.0 171044  2640 pts/3    S    09:25   0:00  |           \_ sudo -s
root      7620  0.0  0.0 108436  2044 pts/3    S+   09:25   0:00  |               \_ /bin/bash
root      8400  0.0  0.0 102544  4512 ?        Ss   09:31   0:00  \_ sshd: mjimeneznet [priv]
500       8443  0.0  0.0 102544  2040 ?        S    09:31   0:00      \_ sshd: [email protected]/4
500       8445  0.0  0.0 110520  1936 pts/4    Ss   09:31   0:00          \_ -bash
root      8484  0.0  0.0 171044  2640 pts/4    S    09:31   0:00              \_ sudo -s
root      8486  0.0  0.0 108436  1984 pts/4    S    09:31   0:00                  \_ /bin/bash
root     26382  0.0  0.0 100948   652 pts/4    S+   10:03   0:00                      \_ tail -f /var/log/messages
root       990  0.0  0.0  21720   608 ?        Ss   Dec01   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root      1248  0.0  0.0  66444   284 ?        Ss   Dec01   0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
root      1249  0.0  0.0  66444    32 ?        S    Dec01   0:00  \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
root      1358  0.0  0.0 138660  1776 ?        Ss   Dec01   0:00 pure-ftpd (SERVER)
554      18726  0.0  0.0 138936  2112 ?        S    09:55   0:00  \_ pure-ftpd (IDLE)
root     18727  0.0  0.0 138664   544 ?        S    09:55   0:00  |   \_ pure-ftpd (PRIV)
554      21839  0.0  0.0 138820  1192 ?        S    09:57   0:00  \_ pure-ftpd (IDLE)
root     21840  0.0  0.0 138664   544 ?        S    09:57   0:00  |   \_ pure-ftpd (PRIV)
554      22216  0.0  0.0 138804  2048 ?        S    09:57   0:00  \_ pure-ftpd (IDLE)
root     22217  0.0  0.0 138664   544 ?        S    09:57   0:00  |   \_ pure-ftpd (PRIV)
547      26242  0.0  0.0 138664  1012 ?        S    10:03   0:00  \_ pure-ftpd (IDLE)
root     26248  0.0  0.0 138664   544 ?        S    10:03   0:00  |   \_ pure-ftpd (PRIV)
547      26319  0.0  0.0 139204  1024 ?        S    10:03   0:00  \_ pure-ftpd (IDLE)
root     26323  0.0  0.0 138664   544 ?        S    10:03   0:00      \_ pure-ftpd (PRIV)
root      1360  0.0  0.0 136148  1392 ?        S    Dec01   0:00 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/local/cpanel/bin/pureauth
root      1372  0.0  0.0 116872   772 ?        Ss   Dec01   0:00 crond
root      1387  0.0  0.0  21112   476 ?        Ss   Dec01   0:00 /usr/sbin/atd
root      1514  0.0  0.1 105468 30100 ?        S    Dec01   0:09 cpsrvd (SSL) - waiting f --llu=1480583561 --listen=10,11,3,4,5,6,7,8,9 --accepted=12
root     23708  0.0  0.1 111596 26284 ?        S    09:58   0:00  \_ whostmgrd - serving 88.1 --llu=1480583561 --listen=10,11,3,4,5,6,7,8,9 --accepted=12
root      1527  0.0  0.0  45544 10092 ?        S    Dec01   0:01 queueprocd - wait to process a task
root      1540  0.0  0.0  26840  3548 ?        Ss   Dec01   0:01 dnsadmin - dormant mode                    --listen=3
root      1554  0.0  0.1  99344 24696 ?        S    Dec01   0:07 tailwatchd
mailman   1560  0.0  0.0 119444  9720 ?        Ss   Dec01   0:00 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/mailmanctl -s start
mailman   1562  0.0  0.0 119256 11696 ?        S    Dec01   0:05  \_ /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=ArchRunner:0:1 -s
mailman   1563  0.0  0.0 119456 11768 ?        S    Dec01   0:06  \_ /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=BounceRunner:0:1 -s
mailman   1564  0.0  0.0 119280 11712 ?        S    Dec01   0:06  \_ /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=CommandRunner:0:1 -s
mailman   1565  0.0  0.0 119288 11708 ?        S    Dec01   0:06  \_ /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s
mailman   1566  0.0  0.0 119316 11768 ?        S    Dec01   0:06  \_ /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=NewsRunner:0:1 -s
mailman   1567  0.0  0.0 119660 11820 ?        S    Dec01   0:06  \_ /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s
mailman   1568  0.0  0.0 119404 11704 ?        S    Dec01   0:05  \_ /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=VirginRunner:0:1 -s
mailman   1569  0.0  0.0 119272 11692 ?        S    Dec01   0:00  \_ /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=RetryRunner:0:1 -s
root      1585  0.0  0.1  81464 18448 ?        S    Dec01   0:05 cPhulkd - processor
root      1597  0.0  0.0  27396  3144 ?        S    Dec01   0:00 cpdavd - accepting connections on 2077, 2078, 2079, and 2080 (dormant)
root      1602  0.0  0.0  25768  2380 ?        SN   Dec01   0:00 cpanellogd - waiting for child to process logs
root     27131  1.7  0.1 110396 31808 ?        DN   10:05   0:00  \_ cpanellogd - seperating xferlog
root      1637  0.0  0.0   4068   576 tty1     Ss+  Dec01   0:00 /sbin/mingetty console
root      1638  0.0  0.0   4068   572 tty2     Ss+  Dec01   0:00 /sbin/mingetty tty2
root     19767  0.4  0.0  26444  3120 ?        Ss   05:25   1:11 spamd-dormant: waiting for connections  --max-spare=1 --max-children=3 --allowed-ips=127.0.0.1,::1 --pidfile=/var/run/spamd.pid --listen=5

 

[Eric]

Install sar and start the service. It'll help you find what's driving up the load.

Thanks!

 

[mjimeneznet]

I have sar installed and what I see from that is that this started on 22 November. No new account added on that week. As I said, everything is stopped and the server continues increasing its load.

I never saw something like that.

 

[cPanelMichael]

Hello,

You may also want to reach out to your provider to determine if there's any issues with the server hardware (e.g. failing hard drive). There's also a third-party thread here you may want to look into:

Hard Drive health check

Thank you.