Overloaded server, load average sometimes over 100.0

[Ramon Pego]

Hello, at about the same time yesterday my server crashed, it was impossible to access the sites hosted on them, or even access the server via whm.foo.com
Today after the same thing happened, I managed to log in to the server after a while and saw that the load average was very high as mentioned above 100.
What really caught my attention was that the 2 were around the same time
11:20 America / Sao_Paulo Time.
I'll send some log files, i don't quite understand them

newest entrys on

/var/log/messages

and

/var/log/dmesg

 

[cPanelLauren]

This looks like a brute force attack but all i see in the logs are Brute Force warnings from cPhulk (prior to removing the 3rd party link) The dmesg output is from reboot on and nothing prior so it's not extremely helpful.

What are you looking for when you're getting the log data from /var/log/messages
If the issue does occur again the lines of the log prior to reboot would be useful but even more useful would be behavior as it occurs. Some of the following may be:

 netstat -plan|egrep 'tcp|udp' |awk '{print $5, $7}'|cut -d: -f1 |sort |uniq -c |sort -n

Also the following may be helpful in identifying the cause Tutorial - Troubleshooting high server loads on Linux servers

 

[Ramon Pego]

This looks like a brute force attack but all i see in the logs are Brute Force warnings from cPhulk (prior to removing the 3rd party link) The dmesg output is from reboot on and nothing prior so it's not extremely helpful.

What are you looking for when you're getting the log data from /var/log/messages
If the issue does occur again the lines of the log prior to reboot would be useful but even more useful would be behavior as it occurs. Some of the following may be:

 netstat -plan|egrep 'tcp|udp' |awk '{print $5, $7}'|cut -d: -f1 |sort |uniq -c |sort -n

Also the following may be helpful in identifying the cause Tutorial - Troubleshooting high server loads on Linux servers

Hello it just happened again, I'm waiting for the server to be accessible to try to recover the logs.
During the fall I can't get access.
I showed these logs at the suggestion of the company that owns the bare metal which i'm using for server, I am relatively new in this type of problem.
First time that occurs to me

I saw that you removed the logs about being third party links, where I can send them? Because here I can only send photos

 

[cPanelLauren]

You can copy paste in the thread within code blocks.

 

[Ramon Pego]

This morning I had an increase in the server load I went to check this command that you mentioned I saw that apparently it has a lot of simultaneous connection.

1 137.175.110.168
      1 176.9.4.102
      1 177.128.87.178
      1 177.22.225.110
      1 216.244.66.194
      1 216.244.66.234
      1 216.244.66.243
      1 45.148.10.85
      1 45.148.10.88
      1 45.148.10.90
      1 45.164.42.103
      1 45.232.156.29
      1 46.229.168.132
      1 46.229.168.135
      1 46.229.168.139
      1 46.229.168.140
      1 46.229.168.141
      1 46.229.168.143
      1 46.229.168.162
      1 46.4.107.106
      1 46.4.108.51
      1 52.252.251.49
      1 52.96.32.125
      1 66.249.79.90
      1 66.249.79.91
      1 69.162.124.229
      1 88.198.17.136
      1 88.99.150.47
      1 94.130.237.173
      1 95.216.172.193
      2 138.59.220.106
      2 164.90.151.179
      2 177.128.87.192
      2 177.74.224.215
      2 192.241.229.214
      2 192.249.126.159
      2 201.162.102.90
      2 222.186.42.137
      2 46.229.168.163
      2 8.8.8.8
      3 174.228.8.251
      3 34.125.197.200
      3 45.148.10.81
      3 45.148.10.87
      3 45.183.10.41
      4 45.164.42.99
      4 45.167.47.65
      6 192.29.97.49
      6 45.231.57.143
      9 45.183.10.34
     10 49.88.112.73
     15 192.241.141.217
     21 167.249.66.7
     40 0.0.0.0
     47 127.0.0.1
     66

cPHulkd is what consumes the most on the server, I know it is a protection of the server.
So I don't understand what to do.
when i check /usr/local/cpanel/logs/cphulkd.log

[2020-08-28 08:08:39 -0300] info [cPhulkd] Login Blocked: The IP address is marked as an excessive brute. [Service]=[sshd] [Remote IP Address]=[222.186.42.7] [Authentication Database]=[system] [Username]=[root] (blocked until [Sat Aug 29 11:08:37 2020 UTC/Sat Aug 29 08:08:37 2020 LOCAL])
[2020-08-28 08:08:41 -0300] info [cPhulkd] Login Blocked: The IP address is marked as an excessive brute. [Service]=[sshd] [Remote IP Address]=[222.186.42.7] [Authentication Database]=[system] [Username]=[root] (blocked until [Sat Aug 29 11:08:37 2020 UTC/Sat Aug 29 08:08:37 2020 LOCAL])
[2020-08-28 08:08:42 -0300] info [cPhulkd] Login Blocked: IP reached maximum auth failures [Service]=[sshd] [Remote IP Address]=[114.67.80.209] [Authentication Database]=[system] [Username]=[arm] (5/5 failures) (blocked until [Fri Aug 28 11:23:42 2020 UTC/Fri Aug 28 08:23:42 2020 LOCAL])
[2020-08-28 08:10:36 -0300] info [cPhulkd] Login Blocked: IP reached maximum auth failures [Service]=[sshd] [Remote IP Address]=[115.159.119.35] [Authentication Database]=[system] [Username]=[root] (5/5 failures) (blocked until[Fri Aug 28 11:25:36 2020 UTC/Fri Aug 28 08:25:36 2020 LOCAL])
[2020-08-28 08:12:31 -0300] info [cPhulkd] Login Blocked: IP reached maximum auth failures [Service]=[sshd] [Remote IP Address]=[96.114.71.147] [Authentication Database]=[system] [Username]=[root] (7/5 failures) (blocked until [Fri Aug 28 11:27:31 2020 UTC/Fri Aug 28 08:27:31 2020 LOCAL])
[2020-08-28 08:18:37 -0300] info [cPhulkd] Login Blocked: IP reached maximum auth failures [Service]=[sshd] [Remote IP Address]=[125.124.97.15] [Authentication Database]=[system] [Username]=[root] (6/5 failures) (blocked until [Fri Aug 28 11:33:37 2020 UTC/Fri Aug 28 08:33:37 2020 LOCAL])
[2020-08-28 08:23:32 -0300] info [cPhulkd] Login Blocked: IP reached maximum auth failures [Service]=[sshd] [Remote IP Address]=[106.12.202.119] [Authentication Database]=[system] [Username]=[webserver] (6/5 failures) (blocked until [Fri Aug 28 11:38:32 2020 UTC/Fri Aug 28 08:38:32 2020 LOCAL])
[2020-08-28 08:25:55 -0300] info [cPhulkd] Login Blocked: IP reached maximum auth failures [Service]=[sshd] [Remote IP Address]=[115.159.119.35] [Authentication Database]=[system] [Username]=[teacher1] (5/5 failures) (blocked until [Fri Aug 28 11:40:55 2020 UTC/Fri Aug 28 08:40:55 2020 LOCAL])
[2020-08-28 08:26:00 -0300] info [cPhulkd] Login Blocked: IP reached maximum auth failures [Service]=[sshd] [Remote IP Address]=[114.67.80.209] [Authentication Database]=[system] [Username]=[ding] (5/5 failures) (blocked until [Fri Aug 28 11:41:00 2020 UTC/Fri Aug 28 08:41:00 2020 LOCAL])
[2020-08-28 08:27:42 -0300] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[96.114.71.147] [Authentication Database]=[system] [Username]=[cognos] (30/30 failures) (blocked until [Sat Aug 29 11:27:42 2020 UTC/Sat Aug 29 08:27:42 2020 LOCAL])

 

[andrew.n]

Login Blocked: IP reached maximum auth failures [Service]=[sshd] [Remote IP Address

It's a brute force attack towards your SSH service. I suggest you to change the SSH port from the default 22 to something else to prevent this abusive behaviour.

 

[keat63]

+1 on changing the port

In /etc/ssh/ssh_config
search for Port
Change the value from #22 or whatever it is to something else

If you can determine who actually requires SSHD, then you could close access to SSHD by using HostAccessControl

Host Access Control | cPanel & WHM Documentation

Use this interface to allow, reject, or drop access to services for specific IP addresses.

docs.cpanel.net

 

[studio triD]

I'm experiencing very heavy server load also.
I can't figure it out, can you help?


Here are some info's on system after restart:

[[email protected] ~]#  netstat -plan|egrep 'tcp|udp' |awk '{print $5, $7}'|cut -d: -f1 |sort |uniq -c |sort -n
      1 109.245.175.7
      1 111.202.101.68
      1 114.119.133.201
      1 114.119.133.220
      1 114.119.136.78
      1 114.119.137.173
      1 114.119.142.135
      1 114.119.146.251
      1 114.119.148.64
      1 114.119.153.106
      1 114.119.166.212
      1 123.183.224.103
      1 123.183.224.105
      1 151.80.111.155
      1 157.55.39.243
      1 173.91.32.118
      1 185.15.22.168
      1 185.191.171.11
      1 185.191.171.13
      1 185.191.171.20
      1 185.191.171.25
      1 185.191.171.26
      1 185.191.171.3
      1 194.127.178.181
      1 199.16.157.181
      1 207.46.13.41
      1 212.70.149.4
      1 213.133.98.98
      1 213.198.254.134
      1 31.13.127.24
      1 40.77.167.208
      1 49.7.20.141
      1 54.36.148.13
      1 54.36.148.132
      1 54.36.148.188
      1 54.36.148.19
      1 54.36.148.196
      1 54.36.148.197
      1 54.36.148.208
      1 54.36.148.44
      1 54.36.148.5
      1 54.36.148.71
      1 54.36.148.91
      1 54.36.149.104
      1 5.9.17.138
      1 62.240.24.117
      1 62.240.25.47
      1 62.240.30.0
      1 64.233.184.27
      1 66.249.69.221
      1 66.249.75.197
      1 76.94.69.218
      1 77.88.9.131
      1 77.88.9.132
      1 77.88.9.137
      1 81.177.6.117
      1 88.152.185.96
      1 91.115.241.166
      1 93.136.201.208
      1 95.217.145.41
      2 108.20.78.134
      2 109.245.227.27
      2 144.76.162.206
      2 148.63.65.89
      2 173.238.236.163
      2 178.220.212.26
      2 185.191.171.12
      2 185.191.171.16
      2 185.191.171.22
      2 185.191.171.23
      2 185.191.171.33
      2 185.191.171.7
      2 2001
      2 207.46.13.155
      2 207.46.13.2
      2 207.46.13.29
      2 212.34.48.245
      2 34.253.208.243
      2 5.188.84.119
      2 78.46.90.120
      2 82.211.161.133
      2 87.116.177.102
      2 91.232.239.102
      3 178.148.65.103
      3 185.191.171.17
      3 185.191.171.21
      3 207.46.13.96
      3 45.118.145.52
      3 46.101.139.73
      4 185.191.171.10
      4 185.191.171.35
      4 185.191.171.4
      4 185.191.171.8
      4 185.191.171.9
      4 45.32.138.106
      4 95.180.127.169
      5 157.55.39.156
      5 185.191.171.18
      5 185.191.171.19
      5 185.191.171.5
      5 185.191.171.6
      5 34.91.150.112
      5 40.77.167.103
      6 109.239.229.238
      6 157.55.39.178
      6 185.191.171.1
      6 185.191.171.14
      6 185.191.171.24
      6 212.200.181.83
      7 127.0.0.1
      8 77.88.9.136
     17
     20 62.240.24.111
     30 0.0.0.0

[[email protected] ~]# sar -q
Linux 3.10.0-327.22.2.el7.x86_64 ()       09/10/2020      _x86_64_        (8 CPU)

12:00:02 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
12:10:01 AM        17       461      3.54      4.43      5.35         1
12:20:01 AM        15       458      2.36      2.85      4.13         0
12:30:01 AM        14       444      2.78      2.82      3.62         0
12:40:01 AM        12       501      2.65      2.78      3.26         0
12:50:02 AM        17       451      5.10      3.99      3.49         7
01:00:01 AM        19       477      6.28      5.20      4.32         0
01:10:01 AM        19       479      9.46      7.83      5.90         0
01:20:01 AM        10       451      7.73      7.55      6.54         1
01:30:01 AM        18       462      8.90      8.56      7.33         0
01:40:01 AM        17       486     10.64      9.35      8.28         0
01:50:01 AM        23       449      6.51      8.19      8.49         1
02:00:01 AM        20       456     16.07     12.93     10.34         1
02:10:01 AM        15       462      6.79      9.25     10.30         0
02:20:01 AM        18       453     10.72      9.24      9.76         1
02:30:01 AM        24       470      7.31      8.37      9.02         0
02:40:01 AM        14       474      6.85      8.16      9.07         1
02:50:01 AM        24       457      8.90      8.22      8.61         1
03:00:01 AM        25       500      8.96      7.51      7.91         3
03:10:01 AM        18       441      7.04      7.85      8.06         1
03:20:01 AM        17       474     11.60     10.41      9.01         2
03:30:01 AM        24       476     11.18      8.98      8.77         0
03:40:02 AM        19       460      6.64      8.17      8.72         5

03:40:02 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
03:50:02 AM         2       453      5.18      8.97      9.30         6
04:00:01 AM        14       485      7.12      7.16      8.13         3
04:10:01 AM        21       469      7.69      7.10      7.60         2
04:20:02 AM        20       469      5.34      6.00      6.91         7
04:30:01 AM        17       478      5.71      6.05      6.48         3
04:40:01 AM         5       465      5.95      6.55      6.57         2
04:50:01 AM        13       477      7.65      6.81      6.70         6
05:00:01 AM         9       476      5.08      5.59      6.13        15
05:10:02 AM        19       471      7.13      7.31      6.70         3
05:20:01 AM         9       477      6.39      6.83      6.82         4
05:30:01 AM        17       481      6.24      6.02      6.44        11
05:40:01 AM         9       464      8.01      8.42      7.83         1
05:50:01 AM        14       479      6.63      6.41      7.05         2
06:00:01 AM        18       469      4.62      8.48      8.31        13
06:10:01 AM        35       505     17.00     11.09      9.40         1
06:20:02 AM         3       481     14.87     17.00     12.28         8
06:30:01 AM        10       447      7.51      8.28      9.66         0
06:40:01 AM        12       467      4.35      5.96      7.81         9
06:50:02 AM        20       488      5.90      5.97      6.92         4
07:00:01 AM         9       492      8.68      9.47      8.13        11
07:10:01 AM        18       476      7.15     10.17      9.38         1
07:20:01 AM        16       471      6.81      6.91      7.86         2

07:20:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
07:30:01 AM         9       473      8.11      7.06      7.40         3
07:40:01 AM        18       474      6.77      6.63      6.90         2
07:50:01 AM        18       478      7.44     10.05      8.62         3
08:00:01 AM        15       482      9.35     11.14      9.65         2
08:10:01 AM        14       490     11.30      8.81      8.81         0
08:20:01 AM        11       457     11.02     12.56     10.87         1
08:30:01 AM        12       473      7.94      8.44      9.49         1
08:40:01 AM        17       482      6.62      6.45      7.86         1
08:50:02 AM        19       486      9.69      9.96      8.80         7
09:00:02 AM        21       534     11.33     11.90     10.60         4
09:10:02 AM        17       522      8.85     11.99     11.30         0
09:20:01 AM        16       491      5.64      7.04      8.95         0
09:30:02 AM         3       467      6.46      6.49      7.82        14
09:40:01 AM         4       496      7.35      6.99      7.45        12
09:50:01 AM         1       472      6.41      6.82      7.34         9
10:00:01 AM        19       485      5.82      6.56      7.07         3
10:10:01 AM         7       462     10.63      8.27      7.45         4
10:20:01 AM        17       486      6.78      7.66      7.63         2
10:30:01 AM        18       500      7.46      7.34      7.40         4
10:40:01 AM        14       508      5.48      6.65      6.98         1
10:50:01 AM         6       503      6.94      6.74      6.94         4
11:00:02 AM         2       541      6.13      6.29      6.63        14

11:00:02 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
11:10:02 AM         1       505      6.37      7.31      7.10        10
11:20:01 AM        21       497      4.45      5.32      6.20         1
11:30:01 AM         5       481     10.14      7.50      6.69         0
11:40:01 AM         5       483      7.92      9.45      8.41        16
11:50:02 AM        24       535     12.24      9.14      8.52         4
12:00:02 PM        38       599     28.07     17.67     12.53        12
12:10:01 PM         3       476      9.32     11.83     12.21         7
12:20:02 PM         2       520      9.28      8.91     10.41         7
12:30:02 PM         8       518      7.53      8.95      9.95        10
12:40:01 PM        16       514      6.98      7.21      8.54         4
12:50:01 PM        11       543      7.33      8.36      8.79         2
01:00:01 PM        29       536      9.34      7.85      8.15         1
01:10:01 PM        17       552      5.92      6.90      7.59         4
01:20:01 PM        21       571      9.63     11.01      9.77         0
01:30:01 PM        22       547      9.66      8.99      9.13         0
01:40:01 PM        19       540     14.00     12.39     10.77         1
01:50:01 PM        19       537     15.65     16.79     14.17         0
02:00:02 PM        17       529     13.68     14.16     13.89        10
02:10:02 PM        49       599     27.10     18.26     15.54         3
02:20:01 PM        18       524      8.01     10.55     12.81         0
02:30:01 PM        21       547     12.64     11.53     12.00         0
02:40:02 PM        23       578     15.38     12.95     12.34         3

02:40:02 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
02:50:01 PM        37       570     17.73     14.14     13.07         0
03:00:02 PM         9       503      7.13     15.31     14.78         5
03:10:01 PM         9       513      7.26      8.01     11.00         5
03:20:01 PM        24       584      7.19      7.16      9.08         0
03:30:01 PM        11       515      6.95      6.67      7.89         1
03:40:01 PM         1       505      6.49      6.39      7.09         8
03:50:01 PM        28       528      5.86      6.25      6.71         2
04:00:01 PM        22       563      8.47      7.80      7.17         6
04:10:01 PM        16       541      5.51      6.16      6.62         4
04:20:01 PM        17       518      6.35      6.57      6.53         0
04:30:02 PM         3       533      6.58     11.29      9.58         6
04:40:01 PM        19       560      5.02      7.25      8.28         0
04:50:01 PM        20       509      7.92      8.88      8.73         2
05:00:01 PM         5       492     10.85      9.03      8.71         1
05:10:02 PM         4       514      6.89      6.69      7.54        10
05:20:01 PM        18       522      4.87      5.95      6.86         3
05:30:01 PM         6       491      6.21      6.08      6.56         2
05:40:01 PM        10       540     19.40     12.40      8.91        13
05:50:01 PM         4       496      5.74     10.37      9.66         1
06:00:02 PM        17       549     24.47     15.04     11.41        11
06:10:01 PM         7       504      7.00      8.46      9.66         1
06:20:02 PM        19       525     11.56     18.56     15.84         0

06:20:02 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
06:30:01 PM        14       533      8.82      8.91     11.71         2
Average:           15       498      8.63      8.68      8.59         4

06:45:42 PM       LINUX RESTART

06:50:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
07:00:09 PM        19       704     91.45     58.72     34.99        31
Average:           19       704     91.45     58.72     34.99        31

07:09:38 PM       LINUX RESTART

07:14:34 PM       LINUX RESTART

top - 19:27:46 up 15 min,  0 users,  load average: 61.98, 61.69, 36.39
Tasks: 462 total,   1 running, 456 sleeping,   0 stopped,   5 zombie
%Cpu(s):  0.3 us,  0.7 sy,  0.1 ni, 19.4 id, 79.5 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32507040 total,   200788 free, 28326732 used,  3979520 buff/cache
KiB Swap: 16760828 total, 13841892 free,  2918936 used.   265600 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
7597 nobody    20   0  309516  28240  14332 D   1.3  0.1   0:00.98 /opt/cpanel/ea-php56/root/usr/bin/php-cgi
4391 mysql     20   0 4565992 174340      0 S   0.7  0.5   1:15.85 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=cp.pro+
5382 munin     30  10  270536  12216   3204 D   0.7  0.0   0:02.44 /usr/local/cpanel/3rdparty/perl/530/bin/perl /usr/local/cpanel/3rdparty/share/munin/munin-graph --cron
7774 root      20   0  160512   2472   1408 R   0.7  0.0   0:00.06 top c
   92 root      20   0       0      0      0 D   0.3  0.0   0:03.25 [kswapd0]
  502 root      20   0       0      0      0 S   0.3  0.0   0:00.50 [md2_raid1]
2705 root      20   0   70604   6520   1564 D   0.3  0.0   0:00.76 tailwatchd
2851 root      20   0 1417196  39612    720 S   0.3  0.1   0:00.56 /usr/local/cpanel/3rdparty/bin/clamd
6200 root      20   0  254724   1708    700 S   0.3  0.0   0:00.57 /usr/sbin/httpd -k start
7543 damdam    20   0  441116  32332   2408 D   0.3  0.1   0:00.53 php-fpm: pool kvaka22_com
7613 root      20   0  416172   1324    104 D   0.3  0.0   0:00.53 php-fpm: master process (/opt/cpanel/ea-php56/root/etc/php-fpm.conf)
7863 nobody    20   0  274180  15536    572 S   0.3  0.0   0:00.01 /usr/sbin/httpd -k start
    1 root      20   0  191564   1264    356 D   0.0  0.0   0:02.62 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 [kthreadd]
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.00 [ksoftirqd/0]
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 [kworker/0:0H]
    7 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 [migration/0]

top - 19:36:32 up 24 min,  0 users,  load average: 181.01, 127.14, 74.91
Tasks: 608 total,   1 running, 606 sleeping,   0 stopped,   1 zombie
%Cpu(s):  2.1 us,  1.0 sy,  0.2 ni,  0.1 id, 96.6 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32507040 total,  2437032 free, 24994948 used,  5075060 buff/cache
KiB Swap: 16760828 total, 11533212 free,  5227616 used.  3331204 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
8819 nobody    20   0  357800  40188  26076 D   2.3  0.1   0:00.34 php-cgi
8913 nobody    20   0  357800  40180  26076 D   2.3  0.1   0:00.31 php-cgi
8914 nobody    20   0  357800  40184  26076 D   2.3  0.1   0:00.31 php-cgi
8919 nobody    20   0  357800  40188  26076 D   2.3  0.1   0:00.31 php-cgi
8899 nobody    20   0  357820  40180  26076 D   2.0  0.1   0:00.34 php-cgi
4725 root      20   0  177496   6928   1200 D   1.3  0.0   0:01.11 lfd - scanning
5382 munin     30  10  270484  12420   3140 D   1.0  0.0   0:03.02 munin-graph
8894 sadanic+  20   0  521984  41444   4864 S   1.0  0.1   0:00.35 php-fpm
9170 studiot+  20   0  460416  53224   4048 D   0.7  0.2   0:00.20 php-fpm
   25 root      20   0       0      0      0 S   0.3  0.0   0:02.18 rcu_sched
   28 root      20   0       0      0      0 S   0.3  0.0   0:01.90 rcuos/2
1572 root      20   0  572396    932    408 S   0.3  0.0   0:00.21 tuned
2392 cpanels+  20   0 6108608  83620   4292 S   0.3  0.3   0:14.02 java
4391 mysql     20   0 4567048 222084   2212 S   0.3  0.7   1:22.39 mysqld
8822 nobody    20   0  281348  28980   3188 S   0.3  0.1   0:00.01 httpd
9096 restart+  20   0  490092  51532  15700 D   0.3  0.2   0:00.20 php-fpm

 

[studio triD]

Load Average is close to 300 now!

top - 19:45:30 up 33 min,  0 users,  load average: 290.24, 251.13, 159.28
Tasks: 689 total,   1 running, 683 sleeping,   0 stopped,   5 zombie
%Cpu(s):  1.0 us,  0.3 sy,  0.0 ni,  0.3 id, 98.4 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32507040 total,   927372 free, 27379488 used,  4200180 buff/cache
KiB Swap: 16760828 total, 10658384 free,  6102444 used.  1490500 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
9330 nobody    20   0  315920  63300  44820 D   1.0  0.2   0:00.62 php-cgi
9742 studiot+  20   0  460556  52632   3532 D   1.0  0.2   0:00.32 php-fpm
10041 betapack  20   0  491632  41352   4080 D   1.0  0.1   0:00.16 php-fpm
4391 mysql     20   0 4569424 240572      0 S   0.7  0.7   1:38.76 mysqld
9646 vsclini+  20   0  347088  38204   2816 D   0.7  0.1   0:00.44 php-fpm
   25 root      20   0       0      0      0 S   0.3  0.0   0:02.66 rcu_sched
   27 root      20   0       0      0      0 S   0.3  0.0   0:00.51 rcuos/1
   28 root      20   0       0      0      0 S   0.3  0.0   0:02.33 rcuos/2
2392 cpanels+  20   0 6108608  88168   2968 S   0.3  0.3   0:24.63 java
7447 nobody    20   0  801480 455928  64608 D   0.3  1.4   0:04.04 php-cgi
8911 nobody    20   0  371220  67804  40748 S   0.3  0.2   0:01.04 php-cgi
9249 nobody    20   0  370348  68340  41260 S   0.3  0.2   0:00.76 php-cgi
9332 nobody    20   0  370348  68132  41048 S   0.3  0.2   0:00.78 php-cgi
9653 iup       20   0  471408  26700   8124 D   0.3  0.1   0:00.32 php-fpm
9691 restart+  20   0  467100  25020  12260 D   0.3  0.1   0:00.24 php-fpm
9855 zenenap+  20   0  340428  32284   3080 D   0.3  0.1   0:00.18 php-fpm
9933 mailnull  20   0   80748   8724   4296 D   0.3  0.0   0:00.01 exim

 

[andrew.n]

You didn't sort the processes by CPU usage so unfortunately we are not able to tell you why....maybe it's an attack? Do you see many connections in WHM under Apache Status?

 

[cPanelLauren]

You server load is high but I would concur that this isn't enough information to tell us why. What would also be more useful than just copying and pasting top would be running

apachectl status

Furthermore high load issues are best handled by a system administrator. You may be able to troubleshoot it on your own we have a guide here: Tutorial - Troubleshooting high server loads on Linux servers

If you're unsure of how to perform any of the steps in that documentation and you do not have a qualified system administrator, you might find one here: System Administration Services

Thanks!

 

[Ramon Pego]

hello, just updating the subject, I disabled Password Authorization on SSH, since I do not access the server via SSH, only by whm. the overload stopped, also "Load Averages" decreased, so it could really be an attack via SSH.
I recommend that those who do not use it do the same or change the door as they said.
The less possibilities of attack on the server, its better

thanks to everyone involved

 

[ScottyBoy]

Not to beat a dead horse, but shodan.io is a great tool to check vulnerabilities and open ports. Your server/IP showed as an open SSH using password auth instead of keys. So yea you were getting brute forced by script kiddies who get the info from shodan which was basically a denial of service as the server could not handle it. Setting up the security is paramount to hosting, else one will get issues like people/bots trying to bruteforce in.

You could also change the settings of openSSH and block the IP after X failed attempts. This is how I have it setup and when I SSH in I can see that thousands have tried to get in, but none were successful and it is a constant attack on my servers, but mitigating configurations like blocking IPs after X Failed attempts is a must if you want to use password SSH (which is not necessarily un-secure, you just need to know the implications)