SOLVED "cPanel & WHM update terminated" (over 6 hours)

mvandemar

Well-Known Member
Jun 17, 2006
133
30
178
Last week one of the servers I maintain ran out of disk space during the backup procedure. I added a new disk and configured the backups to use it instead. They are not retained locally, they are offloaded to Amazon S3 and then removed. There is plenty of space on the new disk.

Every night since I did this though I have been receiving 2 emails back to back. The first one has the subject line, "[vps.domain.com] cPanel & WHM update terminated", and in the body it says this:

Code:
 The system ran the “/usr/local/cpanel/scripts/upcp” script as PID “6818” for longer than 6 hours. The system will terminate this process and run the script again.

Update log preview:
number_of_preview_lines
And attached is a 0 byte update_log.txt. The second email has the subject line, "Cron <[email protected]> (/usr/local/cpanel/scripts/fix-cpanel-perl; /usr/local/cpanel/scripts/upcp --cron > /dev/null)", and this in the body (plaintext, unformatted):

Code:
/bin/sh: line 1:  6818 Killed                  /usr/local/cpanel/scripts/upcp --cron > /dev/null
There are backups being uploaded to Amazon, so that seems to be working fine now, and I am not sure if it's related at all to the update process failing, but it did start right after that incident. If I ssh in and run "ps aux | grep upcp" this is what I see currently:

Code:
[[email protected] ~]# ps aux | grep upcp
root      7638  0.0  0.0 113280  1196 ?        Ss   03:40   0:00 /bin/sh -c (/usr/local/cpanel/scripts/fix-cpanel-perl; /usr/local/cpanel/scripts/upcp --cron > /dev/null)
root      7643  0.0  0.0 113280   568 ?        S    03:40   0:00 /bin/sh -c (/usr/local/cpanel/scripts/fix-cpanel-perl; /usr/local/cpanel/scripts/upcp --cron > /dev/null)
root      7658  0.0  0.2 152344 10772 ?        S    03:40   0:01 cPanel Update (upcp) - Master
root      7662  0.0  0.0      0     0 ?        Z    03:40   0:00 [upcp] <defunct>
root     13718  0.0  0.0 112808   972 pts/0    S+   05:41   0:00 grep --color=auto upcp
Any thoughts where the hangup might be? Thanks.

-Michael
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
7,519
1,028
313
cPanel Access Level
Root Administrator
Hey there! It might be best to kill all of those active upcp processes, and then force an update with "/scripts/upcp --force"

As to the original issue, do you see anything in the logs in /var/cpanel/updatelogs? I know you mentioned you are getting a zero-byte message in the email notification, but there could be older versions that show more detail.
 

mvandemar

Well-Known Member
Jun 17, 2006
133
30
178
As to the original issue, do you see anything in the logs in /var/cpanel/updatelogs? I know you mentioned you are getting a zero-byte message in the email notification, but there could be older versions that show more detail.
The last update log was from 9/29 at 3:40am UTC, and seems to have completed successfully:

Code:
[[email protected] ~]# tail /var/cpanel/updatelogs/update.4679.715695505.1625729847.log
[2021-09-29 03:40:37 +0000]   95% complete
[2021-09-29 03:40:37 +0000]   Running Standardized hooks
[2021-09-29 03:40:37 +0000]   100% complete
[2021-09-29 03:40:37 +0000]   
[2021-09-29 03:40:37 +0000]     cPanel update completed
[2021-09-29 03:40:37 +0000]   A log of this update is available at /var/cpanel/updatelogs/update.4679.715695505.1625729847.log
[2021-09-29 03:40:37 +0000]   Removing upcp pidfile
[2021-09-29 03:40:37 +0000]   
[2021-09-29 03:40:37 +0000] Completed all updates
=> Log closed Wed Sep 29 03:40:37 2021
Hey there! It might be best to kill all of those active upcp processes, and then force an update with "/scripts/upcp --force"
Trying that now, got this far and appears to just be hanging:

Code:
[[email protected] ~]# /scripts/upcp --force
[2021-10-05 22:19:09 +0000] W previous PID (7658) has been running more than 6 hours. Killing processes.
Use of uninitialized value $file in open at /usr/local/cpanel/Cpanel/LoadFile.pm line 47.
Sending kill signal to process group for 7658
Waiting for processes to die
Done!
Stale PID file '/var/run/upcp.pid' (pid=7658)
[2021-10-05 22:19:10 +0000]   Detected cron=0 (--force passed on command line)
Can't use an undefined value as a symbol reference at /scripts/upcp line 268.
Googling that error in quotes, minus the line number, brings up nothing. Also, I had already killed 7658, not sure why it said it was still running, I confirmed there were no more processes after I killed them all.

-Michael
 

mvandemar

Well-Known Member
Jun 17, 2006
133
30
178
Note: 6 hours later and no change, killing the process. Any thoughts on what to try next?

-Michael
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
7,519
1,028
313
cPanel Access Level
Root Administrator
I don't have any other ideas on my end for that, as I also don't have anything on my side when I search for that particular error.

Could you create a ticket with our team so we can check this directly on the system? If you can do that, just post the ticket number here so I can follow along.
 

mvandemar

Well-Known Member
Jun 17, 2006
133
30
178
@cPRex They were able to fix it by running /scripts/rpmup first, and then /scripts/upcp --force ran just fine.

-Michael