ksoftirqd/3 high cpu usage

[hrace009]

Hello,

ksoftirqd/3 use high cpu process when cpbackup running. and it make the system running very slow.

top - 14:20:29 up 1 day,  1:01,  2 users,  load average: 3.38, 3.50, 3.35
Tasks: 241 total,   5 running, 233 sleeping,   0 stopped,   3 zombie
%Cpu(s):  2.6 us,  1.9 sy,  0.4 ni, 78.9 id,  0.2 wa,  0.0 hi, 12.4 si,  3.6 st
KiB Mem : 32703208 total,   379576 free,  2145696 used, 30177936 buff/cache
KiB Swap:  8388604 total,  8357716 free,    30888 used. 29846932 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   23 root      20   0       0      0      0 R  72.2  0.0 533:27.29 ksoftirqd/3
13653 unilak    20   0       0      0      0 Z  14.2  0.0   0:00.51 php-cgi
21167 root      38  18  255764  46520   4720 S  13.6  0.1  14:38.08 cpbackup_transp
   22 root      rt   0       0      0      0 S  12.6  0.0  43:52.42 migration/3
 3287 root      20   0       0      0      0 R   7.6  0.0   0:50.51 kworker/0:2
 1273 cpanelc+  20   0   11440   4508   4316 R   6.3  0.0  30:47.09 p0f
12661 unilak    20   0  679876  36052  17244 S   5.0  0.1   0:02.28 php-fpm
13642 unicorn+  20   0  600248  29148  14428 S   4.6  0.1   0:00.17 php-fpm
14697 root      20   0       0      0      0 S   4.3  0.0   1:39.59 kworker/3:1
   21 root      rt   0       0      0      0 S   4.0  0.0  13:16.69 watchdog/3
20542 mysql     20   0 1112956 360640  12544 S   2.3  1.1   5:28.11 mysqld
12881 pwtwili+  20   0  597640  21152   9060 S   1.3  0.1   0:00.40 php-fpm
13632 nobody    20   0  410312 114808   3108 S   1.3  0.4   0:00.06 httpd
    9 root      20   0       0      0      0 S   1.0  0.0   3:48.74 rcu_sched
15781 root      20   0  182376  29628   1924 S   1.0  0.1   0:25.52 lfd - sleeping
12939 unilak    20   0  410272 114880   3196 S   0.7  0.4   0:00.11 httpd
30525 root      20   0  155932   2236   1448 R   0.7  0.0   1:08.13 top
12806 nobody    20   0       0      0      0 Z   0.3  0.0   0:00.12 httpd
12932 nobody    20   0  410388 114940   3224 S   0.3  0.4   0:00.09 httpd

if i killed cpbackup process, system running normal.

[root@svr2 ~]# ps aux | grep cpbackup
root     13912  0.0  0.0 112660   940 pts/1    S+   14:22   0:00 grep --color=auto cpbackup
root     21167 12.7  0.1 255764 46520 ?        SN   12:25  14:56 cpbackup_transporter - Processing next task
[root@svr2 ~]# kill -9 21167

top - 14:23:55 up 1 day,  1:05,  2 users,  load average: 2.00, 2.94, 3.17
Tasks: 222 total,   2 running, 220 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  0.4 sy,  0.0 ni, 86.1 id,  1.5 wa,  0.0 hi, 10.6 si,  0.8 st
KiB Mem : 32678632 total,   418576 free,  1942220 used, 30317836 buff/cache
KiB Swap:  8388604 total,  8357620 free,    30984 used. 30015200 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   23 root      20   0       0      0      0 R  79.5  0.0 535:52.18 ksoftirqd/3
12807 unilak    20   0  410652 115572   3396 S   3.0  0.4   0:00.67 httpd
   21 root      rt   0       0      0      0 S   2.0  0.0  13:20.22 watchdog/3
13995 unilak    20   0  548172  41896  26888 S   1.3  0.1   0:00.41 php-cgi
13798 nobody    20   0  410404 115000   3312 S   1.0  0.4   0:00.43 httpd
 1273 cpanelc+  20   0   11440   4508   4316 S   0.7  0.0  30:53.98 p0f
13985 pwtwili+  20   0  597508  19736   8684 S   0.7  0.1   0:00.19 php-fpm
20542 mysql     20   0 1112956 360444  12548 S   0.7  1.1   5:31.29 mysqld
    9 root      20   0       0      0      0 S   0.3  0.0   3:49.64 rcu_sched
30525 root      20   0  155932   2236   1448 R   0.3  0.0   1:09.18 top
    1 root      20   0  192052   4564   2144 S   0.0  0.0   2:38.72 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.09 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:05.56 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      rt   0       0      0      0 S   0.0  0.0   0:07.58 migration/0
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
   10 root      rt   0       0      0      0 S   0.0  0.0   0:06.57 watchdog/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:02.58 watchdog/1
   12 root      rt   0       0      0      0 S   0.0  0.0   0:09.15 migration/1
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.96 ksoftirqd/1
   15 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   16 root      rt   0       0      0      0 S   0.0  0.0   0:03.06 watchdog/2
   17 root      rt   0       0      0      0 S   0.0  0.0   0:07.91 migration/2
   18 root      20   0       0      0      0 S   0.0  0.0   0:00.64 ksoftirqd/2

is there a way to fix this?

at this time, i have turn off automatic backup to avoid this problem.

 

[cPanelMichael]

Hello,

Do you happen to have backups configured on a network device, or secondary hard drive? What CPU/RAM is utilized on this server? Is this new behavior, or is this your first time enabling backups?

Thank you.

 

[hrace009]

Hello,

Do you happen to have backups configured on a network device, or secondary hard drive?

Thank you.

i configure backup with google drive

Hello,

What CPU/RAM is utilized on this server?

Thank you.

for cpbackup it doesn't make CPU/RAM high usage. but when cpbackup run, ksoftirqd/3 become high CPU utililzed

Hello,

Is this new behavior, or is this your first time enabling backups?

Thank you.

i have enable backup with google drive since it release on cpanel, and never have this problem.

 

[cPanelMichael]

Hello,

Here's part of a response from one of our Technical Analysts on support ticket with a similar inquiry:

ksoftirqd is actually a kernel process, it does not actually use any CPU processing time (thus does not cause the server load to increase). Common tools such as top see it as using CPU time because of how it operates. ksoftirqd is responsible for preventing each CPU core from skipping instructions, which is why there is always one more running than the number of cores a processor has.

The more "individual" ksoftirqd processes you see active means that more of the CPU cores are being used. For example, is an application is designed to use 2 cores, you will see 2 of these "using" CPU time while it manages the interrupts.

I recommend reviewing the following page:

ksoftirqd(9) — Linux manual pages

I highly recommend working with a qualified system administrator if you feel your server is having issues with high load so you can track down the cause and work to mitigate it. If you do not have a dedicated system administrator, cPanel provides a list of system administration services. Please note, cPanel does not endorse or sponsor any service listed on this page.

Resources | cPanel Forums

Thank you.

 

[hrace009]

well, this problem occur from cpanelbackup_transporter / /usr/local/cpanel/bin/backup --force, when it running, it make ksoftirqd/3 using high CPU.
but when i killed the process, ksoftirqd return normal.
so i think the problem is from cPanel it self.

I hope a solution from cPanel team, because the problem from cPanel Backup, and please don't send me to Resources | cPanel Forums
I have send the ticket before, but the support give the same answer. And it hurts my feeling.

 

[cPanelMichael]

Hello,

Could you let us know the ticket number so we can take a closer look?

Thank you.

 

[hrace009]

actually i have closed the ticket, because seems the support don't care with my problem.
I will reopen it for you to investigate it. Ticket number #9168227


the process cpbackup_transport run since 2 AM GMT+7.
And the process never finish until i reply this topic

just FYI, with process like this, it take long time to enter console trough putty.
When i killed the process, server runs normal.

When i look at google drive folder, the file transfer never complete. and it only show 1 files


it should show a few files, but at this time it only show 1 files

 

[cPanelMichael]

Hello,

To update, it looks like you closed the ticket and reported the issue solved after adjusting your CSF firewall configuration (removing IPs from the blacklist).

Thank you.