High Loads daily at the same times

I am monitoring our WHM/Cpanel VPS and notice there are a lot of High Loads during the day.
Most of them show up every day at the same time.

The VPS is a 16 CPU system on a hardware node that has several VPS's running. Not sure how much.

Specific those High Loads are occuring every three hours. But some of them seem to be on other times too, but still daily. Seen peaks of 9.00 Loads for almost 10 minutes of time. Sites are very slow and all WHM/Cpanel functions (webmail etc) are extremely slow during the peaks.

We changed some settings in LFD so every Peak of above two will be reported.

Strange thing is that nothing can be seen during the High Loads, no process is showing that is taking the CPU resources. I checked both, during the Peak via SSH shell (using ps, netstat, vmstat, mpstat) and the logs that are sent by LFD.

Also tried to disable crond for some hours, statistics and backups are only scheduled during the night. Even shut-sown Apache during a one of the High Loads. No effect at all.

What can cause those High Loads?
Why isn't there any process showing up that is consuming all those resources?
How to investigate this further?

Any help is appreciated.

 

The VPS provider cannot find the cause.

The High Loads are there every 90 minutes.
We tried:
- Stopped service cron
- Stopped service LFD/CSF

Why isn't anything showing up with a 'ps auxf' during the High Loads ??
What can be the cause of this?

 

High Loads are there:
11:33:43 PM load average alert - 4.80
10:03:16 PM load average alert - 3.03
08:33:06 PM load average alert - 3.69
07:03:39 PM load average alert - 4.47
05:32:21 PM load average alert - 3.50
04:02:56 PM load average alert - 5.00
02:33:05 PM load average alert - 3.41
01:02:29 PM load average alert - 4.18
11:32:49 AM load average alert - 4.19
10:01:48 AM load average alert - 3.42
08:32:28 AM load average alert - 3.05
07:05:57 AM load average alert - 3.35
05:31:44 AM load average alert - 3.12
04:04:36 AM load average alert - 3.28

Those Loads take around 8 to 12 minutes before getting below 1.00 again. All other times (in between) the Loads are between 0.08 and sometimes peak to 0.8.


Herr the results from the 1PM load:

Output from ps:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 1716 604 ? Ss 2010 0:02 init [3]
root 7604 0.0 0.1 1620 528 ? Ss 2010 0:28 syslogd -m 0
root 7608 0.0 0.0 1568 372 ? Ss 2010 0:00 klogd -x
root 7673 0.0 0.1 4132 1040 ? Ss 2010 0:00 /usr/sbin/sshd
root 7687 0.0 0.1 2188 792 ? Ss 2010 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root 11559 0.0 0.0 1580 408 ? Ss 2010 0:00 /usr/sbin/portsentry -tcp
root 15861 0.0 0.2 7104 1340 ? Ss 2010 0:02 pure-ftpd (SERVER)
root 15891 0.0 0.1 6820 1004 ? S 2010 0:00 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureauth
named 19856 0.0 1.0 195640 5404 ? Ssl 2010 2:55 /usr/sbin/named -u named
mailnull 5817 0.0 0.3 7544 2068 ? Ss 2010 0:01 /usr/sbin/exim -bd -q60m
root 11439 0.0 6.3 35764 33412 ? Ss 2010 0:05 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=3 --max-spare=1
root 3441 0.4 13.0 70572 68512 ? S 10:04 0:51 \_ spamd child
root 7868 0.0 0.8 6172 4644 ? S 2010 0:00 queueprocd - wait to process a task
root 8121 0.0 1.0 7900 5464 ? S 2010 0:05 tailwatchd
root 19762 0.1 1.1 8808 5796 ? S 13:01 0:00 \_ tailwatchd
root 27900 1.2 0.7 5464 3852 ? D 13:02 0:00 \_ /usr/bin/perl /scripts/restartsrv_mysql --check
root 8191 0.0 0.4 4096 2372 ? S 2010 0:00 cPhulkd - processor
root 10122 0.0 1.7 13952 9068 ? S 2010 0:00 cpdavd - accepting connections on 2077 and 2078
root 11338 0.0 0.3 3896 1632 ? SN 2010 0:00 cpanellogd - sleeping for logs
root 13334 0.0 1.4 13440 7748 ? S 2010 0:11 cpsrvd - waiting for connections
root 17795 0.0 0.1 1944 676 ? Ss 2010 0:07 /usr/sbin/dovecot
root 17796 0.0 0.1 2372 1016 ? S 2010 0:08 \_ dovecot-auth
dovecot 18095 0.0 0.3 4100 2056 ? S 2010 0:13 \_ pop3-login
dovecot 18097 0.0 0.3 4100 2012 ? S 2010 0:01 \_ pop3-login
dovecot 18098 0.0 0.3 4072 2056 ? S 2010 0:05 \_ imap-login
dovecot 18099 0.0 0.3 3980 1888 ? S 2010 0:00 \_ imap-login
root 30434 0.0 0.1 5764 1048 ? Ss Jan02 0:00 crond
root 16256 0.0 1.3 12792 6816 ? Ss Jan02 0:08 /usr/local/apache/bin/httpd -k start -DSSL
root 15944 0.0 1.0 8120 5664 ? S 12:07 0:00 \_ /usr/bin/perl /usr/local/cpanel/bin/leechprotect
nobody 3573 0.0 1.2 12928 6576 ? S 12:28 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 26348 0.0 1.2 12928 6484 ? S 12:38 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 22133 0.0 1.2 12928 6456 ? S 12:40 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 25712 0.0 1.2 12928 6388 ? S 12:45 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 11627 0.0 1.2 12928 6572 ? S 12:46 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
italiew 17776 0.7 3.3 32144 17372 ? S 13:01 0:00 | \_ /usr/bin/php
nobody 26195 0.0 1.2 12928 6372 ? S 12:50 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 26403 0.0 1.2 12928 6456 ? S 12:50 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 9323 0.0 1.2 12928 6428 ? S 12:52 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 3529 0.0 1.2 12928 6552 ? S 12:53 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 32618 0.0 1.2 12928 6344 ? S 13:00 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
nobody 27941 0.0 1.0 12792 5448 ? S 13:02 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
root 18312 0.0 0.2 5392 1244 ? S Jan02 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/servername.pid
mysql 18383 0.1 4.7 134500 24832 ? Sl Jan02 5:33 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/servername.pid --skip-external-locking --port=3306 --socket=/var/lib/mysql/mysql.sock
root 28089 0.0 1.9 15396 10448 ? Ss 00:00 0:11 lfd - sleeping
root 7297 0.0 1.7 15396 9424 ? S 13:00 0:00 \_ lfd - checking system integrity
root 7298 0.2 0.1 4708 536 ? D 13:00 0:00 | \_ /usr/bin/md5sum --check /etc/csf/csf.tempint
root 17757 0.1 1.8 15528 9880 ? D 13:01 0:00 \_ lfd - checking directories
root 28254 0.0 1.8 15396 9588 ? S 13:02 0:00 \_ lfd - (child) checking load...
root 28453 0.0 0.1 2396 796 ? R 13:02 0:00 \_ /bin/ps axuf



Output from vmstat:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 4 0 318736 0 0 0 0 56 15 0 1 0 0 97 3




I cannot see the process that is causing the High Load, does this mean another VPS on the node is causing this every 90 minutes? So are the Load values shown in WHM, or when using shell, are showing the loads of all VPS'ses on the node? Why are the loads shown 5.00 or even up to 12.00 when there is no process on the VPS peaking?