cPanel upgrade hanging on dcpumon

Bretas

Active Member
PartnerNOC
Jun 10, 2018
27
5
3
Brazil
cPanel Access Level
Root Administrator
Hello, everyone!

I've been experiencing an issue with one of our servers since May 17th: the process dcpumon keeps hanging indefinitely during cPanel update and eventually is killed. The alert e-mails I'm getting every day ever since contain the following text:

Code:
Update log preview:
...
...
[2018-06-10 03:51:59 -0300] E [/usr/local/cpanel/bin/dcpumon] The “/usr/local/cpanel/bin/dcpumon” command (process 74271) ended prematurely because it received the “KILL” (9) signal.
...
[2018-06-10 03:52:07 -0300] [/usr/local/cpanel/scripts/migrate_local_ini_to_php_ini] No local.ini.
[2018-06-10 03:52:07 -0300] [/usr/local/cpanel/scripts/migrate_local_ini_to_php_ini] … done!
[2018-06-10 03:52:07 -0300] - Finished command `/usr/local/cpanel/scripts/migrate_local_ini_to_php_ini --run --verbose` in 1.332 seconds
[2018-06-10 03:52:07 -0300] 92% complete
[2018-06-10 03:52:08 -0300] 93% complete
[2018-06-10 03:52:08 -0300] - Finished in 0.352 seconds
[2018-06-10 03:52:08 -0300] Processing: Ensuring an "Active" MySQL profile is set
[2018-06-10 03:52:08 -0300] - Processing command `/usr/local/cpanel/scripts/check_mysql`
[2018-06-10 03:52:08 -0300] [/usr/local/cpanel/scripts/check_mysql] “check_mysql” will complete in the background (process ID 75644).
[2018-06-10 03:52:08 -0300] - Finished command `/usr/local/cpanel/scripts/check_mysql` in 0.497 seconds
[2018-06-10 03:52:08 -0300] 94% complete
[2018-06-10 03:52:08 -0300] Processing: Checking CloudLinux installation
[2018-06-10 03:52:08 -0300] - Processing command `/usr/local/cpanel/bin/cloudlinux_update`
[2018-06-10 03:52:11 -0300] - Finished command `/usr/local/cpanel/bin/cloudlinux_update` in 3.004 seconds
[2018-06-10 03:52:11 -0300] Processing: Updating plugins data cache
[2018-06-10 03:52:11 -0300] - Processing command `/usr/local/cpanel/bin/refresh_plugin_cache`
[2018-06-10 03:52:12 -0300] - Finished command `/usr/local/cpanel/bin/refresh_plugin_cache` in 0.640 seconds
[2018-06-10 03:52:12 -0300] 95% complete
=> Log closed Sun Jun 10 03:52:12 2018
----------------------------------------------------------------------------------------------------
=> Log opened from cPanel Update (upcp) - Slave (72346) at Sun Jun 10 03:52:12 2018

Here is what I see in the upgrade log:

Code:
[2018-06-10 01:53:57 -0300]    - Finished in 0.002 seconds
[2018-06-10 01:53:57 -0300]    - Finished in 0.000 seconds
[2018-06-10 01:53:57 -0300]    - Finished in 0.000 seconds
[2018-06-10 01:53:57 -0300]    - Finished in 0.000 seconds
[2018-06-10 01:53:57 -0300]    - Finished in 0.782 seconds
[2018-06-10 01:53:58 -0300]    - Finished in 0.003 seconds
[2018-06-10 01:53:58 -0300]   Processing: Cleaning SpamAssassin DBM files
[2018-06-10 01:53:58 -0300]    - Processing command `/usr/local/cpanel/scripts/spamassassin_dbm_cleaner`
[2018-06-10 01:53:58 -0300]      [/usr/local/cpanel/scripts/spamassassin_dbm_cleaner] /usr/local/cpanel/scripts/spamassassin_dbm_cleaner: version 1.0
[2018-06-10 01:53:59 -0300]      [/usr/local/cpanel/scripts/spamassassin_dbm_cleaner] Checking SpamAssassin dbm databases....Done
[2018-06-10 01:53:59 -0300]      [/usr/local/cpanel/scripts/spamassassin_dbm_cleaner] Checked 16 files for 14 user(s), 16 ok, 0 broken
[2018-06-10 01:53:59 -0300]    - Finished command `/usr/local/cpanel/scripts/spamassassin_dbm_cleaner` in 1.093 seconds
[2018-06-10 01:53:59 -0300]    - Finished in 0.542 seconds
=> Log closed Sun Jun 10 01:53:59 2018
[2018-06-10 01:53:59 -0300]    - Finished in 0.192 seconds
[2018-06-10 01:53:59 -0300]   Processing: Checking for new security advice
[2018-06-10 01:53:59 -0300]    - Processing command `/usr/local/cpanel/scripts/check_security_advice_changes --notify`
[2018-06-10 01:54:36 -0300]      [/usr/local/cpanel/scripts/check_security_advice_changes] There are no changes to the Security Advisor state that require notification.
[2018-06-10 01:54:36 -0300]    - Finished command `/usr/local/cpanel/scripts/check_security_advice_changes --notify` in 37.078 seconds
[2018-06-10 01:54:36 -0300]   Processing: Running former postinstall scripts
[2018-06-10 01:54:36 -0300]    - Processing command `/usr/local/cpanel/bin/dcpumon --killproc`
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/BitchX.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/bnc.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/eggdrop.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/generic-sniffers.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/guardservices.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/ircd.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/psyBNC.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/ptlink.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Loading Symbol Table... /usr/local/cpanel/etc/sym/services.sym  ..Done
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Kill Proc Enabled
[2018-06-10 01:54:37 -0300]      [/usr/local/cpanel/bin/dcpumon] Added Contact [email protected][our_domain]
It won't get any further from there.

When attempting to run the command directly, dcpumon doesn't get stuck or fails at all:

Code:
[[email protected] ~]$ /usr/local/cpanel/bin/dcpumon
Added Contact [email protected][our_domain]
[[email protected] ~]$ echo $?
0
[[email protected] ~]$
I tried temporarily disabling the server's firewall and ran the upgrade again, both forced and otherwise, to no avail.

The box runs on CloudLinux 6 and cPanel version is currently v70.0.48 (which comes to show that the upgrade is indeed working despite the aforementioned error).

Nothing relevant comes up on Google.

Is anyone else experiencing something similar? Could it be some sort of cosmetic error that can be ignored?

Thanks!
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,297
1,251
313
Houston
Hi @Bretas

I haven't seen this specific issue come across in any recent cases or tickets. If you run
Code:
/scripts/dcpumon-wrapper
does it also complete successfully? For some reason when the update completes it' looks like it's not getting the output it's expecting untill the process is killed. It does complete successfully though.
 

Bretas

Active Member
PartnerNOC
Jun 10, 2018
27
5
3
Brazil
cPanel Access Level
Root Administrator
PS: This forum is not allowing me to edit the post above to include the following text ("Your content can not be submitted. This is likely because your content is spam-like"...). So, sorry for posting twice in a row:

I guess I didn't include in my original post the first line (the red one) of the error message we are being emailed. It mentions that the exit code is 256, which is why I keep checking the exit from these commands, I'm trying to reproduce the error cPanel is getting during the update so we can narrow down the root cause. Here it is:

Code:
     Maintenance ended; however, it did not exit cleanly (256). The following events were logged: “bin/dcpumon”. Review the update logs to determine why the update failed.
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,297
1,251
313
Houston
Hi @Bretas

Thank you for getting back to me with that output. Both commands essentially do the same thing which is why I was curious. At this point, I'd like to see if you could please open a ticket using the link in my signature. Once open please post the ticket ID here so I can update this thread with the outcome.

Thanks!
 

jsilvestre

Member
Jul 7, 2005
24
0
151
Hello,

It's a know issue and have an internal case CPANEL-21205 (since today).

As workaround, please disable Background Process Killer.

Best Regards,
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,297
1,251
313
Houston
That's correct, actually as a result of @Bretas ticket we found that the background process killer was consuming a large amount of memory resulting in the process hanging. We have only had 2 reports thus far of this occurring but are hoping to add some more examples to the case. If you are experiencing this issue and have not opened a ticket as of yet please reply to this thread so I can note it in the internal case. If you're unsure you're experiencing this issue and would like for us to take a closer look, please use the link in my signature to open a ticket with us and let us know the ticket ID here.


Thanks!
 

morrow95

Well-Known Member
Oct 8, 2006
161
8
168
I contacted support yesterday for this same issue and support told me there was an open case already for it. CPANEL-21205

If it helps at all my support ticket is/was number 9747109.
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,297
1,251
313
Houston
Hi @morrow95

Thanks, I just double checked the case and found that your ticket was added to it as an example. Thank you for letting us know!


I also see that the case has been updated to indicate that they are testing a fix for it currently. I'll update here again when the case is marked as completed.


Thanks!