Backups not completing since April 8

Kevin C

Member
Apr 10, 2014
6
0
1
cPanel Access Level
Root Administrator
Something in one of the accounts on the server is killing the backup process. On April 8, the backups did not complete, failing at this account. When killing those backup processes and re-running manually and tailing the logfile, we see it again stop on this particular account;

Code:
Copy userdata...Done
Copy custom virtualhost templates...Done
Leaving timeout safety mode
Copying mailman lists and archives....
Done
Copying homedir.............
.........
.........
.........
.........
.........
.........
.........
.........
.........
.........
.........
.........
.........
.........
Done
Calculating nobody files............
Done
Entering timeout safety mode
Determining mysql dbs......mysql version: 5.5......mysqldump version: 5.5...Saving mysql privs...Done
It has been sitting on this step for 8 hours now and the account only has one database at 20MB so it isn't a size issue. I'm not sure where else to look for a solution to this problem.

Any help?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,227
463
Hello :)

Please run the following command when the restoration is hanging:

Code:
ps auxf
Could you verify which particular process that is hanging?

Thank you.
 

Kevin C

Member
Apr 10, 2014
6
0
1
cPanel Access Level
Root Administrator
Code:
root     11663  0.0  0.1 180704 26344 ?        SNs  09:27   0:00 /usr/local/cpanel/bin/backup
root     13889  0.0  0.0   4056   496 ?        SN   09:53   0:00  \_ /usr/local/cpanel/bin/cpuwatch 8.0 /usr/local/cpanel/scripts/pkgacct user /backup/2014-04-10/accounts backup
root     13890  0.0  0.3 254932 55140 ?        DNs  09:53   0:00      \_ pkgacct - user - av: 3
'user' above is being used to mask the actual username.
 

Kevin C

Member
Apr 10, 2014
6
0
1
cPanel Access Level
Root Administrator
Code:
root     11663  0.0  0.1 180704 26344 ?        SNs  09:27   0:00 /usr/local/cpanel/bin/backup
root     13889  0.0  0.0   4056   496 ?        SN   09:53   0:00  \_ /usr/local/cpanel/bin/cpuwatch 8.0 /usr/local/cpanel/scripts/pkgacct user /backup/2014-04-10/accounts backup
root     13890  0.0  0.3 254932 55140 ?        DNs  09:53   0:00      \_ pkgacct - user - av: 3
'user' above is being used to mask the actual username.
Backups still failing this morning

The backup was not able to be completed because timed out waiting for /bin/backup to finish
Really could use some help here, cPanel.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,227
463
Feel free to open a support ticket so we can take a closer look. You can post the ticket number here so we can update this thread with the outcome.

Thank you.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,227
463
To update, it was determined the issue was related to the NFS mount itself. Processes utilizing the NFS mount were instantly placed in the "D" state.

Thank you.
 

keywaystab

Member
May 26, 2005
6
0
151
To update, it was determined the issue was related to the NFS mount itself. Processes utilizing the NFS mount were instantly placed in the "D" state.
Could you post the solution? Was it related to the mount options? Or trouble with the NFS server?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,227
463
The backup process was instantly going into a "D" state when attempting to read/write data on the NFS mount. The customer did not report what method or steps they took to resolve the issue, but it was determined not to be related to the cPanel backup script.

Thank you.
 

noisetheory

Registered
Aug 23, 2004
2
0
151
I'm wondering if anyone has any ideas on the solution here.

I am having the same problem with an NFS mount and it appears the backup process goes to D state immediately when started (though previous days logs show the backup process stopping on a specific account on the same mysql line as posted earlier in the thread).

i'm not an nfs expert and have tried soft mounts, intr option but no luck. While the backup process fails I can successfully copy various files via command line to the nfs mounted location.

Suggestions welcome :)
 

danny0n

Member
Jan 21, 2010
8
0
51
Did you find any solution for this problem ?

I have the same problem on one of my servers. What is weird is that I have a second cpanel server with the same configuration, same NFS server and everything works fine there.

Thank you.
 

danny0n

Member
Jan 21, 2010
8
0
51
The problem seems to be with the horde database.

More specifically the process dies at line 1103 of pkgacct script

Code:
         eval { $horde->backup( $user, $horde_dir, $is_incremental ) };
For the moment I can't find that horde script.
If someone can point me to the script I could investigate this further.

Thank you.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,227
463
The problem seems to be with the horde database.

More specifically the process dies at line 1103 of pkgacct script

Code:
         eval { $horde->backup( $user, $horde_dir, $is_incremental ) };
For the moment I can't find that horde script.
If someone can point me to the script I could investigate this further.
Can you backup the horde database manually? EX:

Code:
mysqldump horde > /path/to/horde_backup.sql
Thank you.
 

danny0n

Member
Jan 21, 2010
8
0
51
Yes, mysqldump works as expected.
It would have been very weird if it didn't.

This problem only occurs when using a NFS for backups.
So probably is something weird happening in the horde backup script that kills the process and I don't think is database related. It works when there is no horde database probably because it skips everything.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,909
2,227
463
Feel free to open a support ticket if you believe this is an issue with cPanel/WHM and not with your NFS mount. You can post the ticket number here so we can update this thread with the outcome.

Thank you.