Backups not completing since April 8

[Kevin C]

Something in one of the accounts on the server is killing the backup process. On April 8, the backups did not complete, failing at this account. When killing those backup processes and re-running manually and tailing the logfile, we see it again stop on this particular account;

Copy userdata...Done
Copy custom virtualhost templates...Done
Leaving timeout safety mode
Copying mailman lists and archives....
Done
Copying homedir.............
.........
.........
.........
.........
.........
.........
.........
.........
.........
.........
.........
.........
.........
.........
Done
Calculating nobody files............
Done
Entering timeout safety mode
Determining mysql dbs......mysql version: 5.5......mysqldump version: 5.5...Saving mysql privs...Done

It has been sitting on this step for 8 hours now and the account only has one database at 20MB so it isn't a size issue. I'm not sure where else to look for a solution to this problem.

Any help?

 

[cPanelMichael]

Hello

Please run the following command when the restoration is hanging:

ps auxf

Could you verify which particular process that is hanging?

Thank you.

 

[Kevin C]

root     11663  0.0  0.1 180704 26344 ?        SNs  09:27   0:00 /usr/local/cpanel/bin/backup
root     13889  0.0  0.0   4056   496 ?        SN   09:53   0:00  \_ /usr/local/cpanel/bin/cpuwatch 8.0 /usr/local/cpanel/scripts/pkgacct user /backup/2014-04-10/accounts backup
root     13890  0.0  0.3 254932 55140 ?        DNs  09:53   0:00      \_ pkgacct - user - av: 3

'user' above is being used to mask the actual username.

 

[Kevin C]

root     11663  0.0  0.1 180704 26344 ?        SNs  09:27   0:00 /usr/local/cpanel/bin/backup
root     13889  0.0  0.0   4056   496 ?        SN   09:53   0:00  \_ /usr/local/cpanel/bin/cpuwatch 8.0 /usr/local/cpanel/scripts/pkgacct user /backup/2014-04-10/accounts backup
root     13890  0.0  0.3 254932 55140 ?        DNs  09:53   0:00      \_ pkgacct - user - av: 3

'user' above is being used to mask the actual username.

Backups still failing this morning

The backup was not able to be completed because timed out waiting for /bin/backup to finish

Really could use some help here, cPanel.

 

[cPanelMichael]

Feel free to open a support ticket so we can take a closer look. You can post the ticket number here so we can update this thread with the outcome.

Thank you.

 

[Kevin C]

Feel free to open a support ticket so we can take a closer look. You can post the ticket number here so we can update this thread with the outcome.

Thank you.

Request ID 4807527

 

[cPanelMichael]

To update, it was determined the issue was related to the NFS mount itself. Processes utilizing the NFS mount were instantly placed in the "D" state.

Thank you.

 

[keywaystab]

To update, it was determined the issue was related to the NFS mount itself. Processes utilizing the NFS mount were instantly placed in the "D" state.

Could you post the solution? Was it related to the mount options? Or trouble with the NFS server?

 

[cPanelMichael]

The backup process was instantly going into a "D" state when attempting to read/write data on the NFS mount. The customer did not report what method or steps they took to resolve the issue, but it was determined not to be related to the cPanel backup script.

Thank you.

 

[noisetheory]

I'm wondering if anyone has any ideas on the solution here.

I am having the same problem with an NFS mount and it appears the backup process goes to D state immediately when started (though previous days logs show the backup process stopping on a specific account on the same mysql line as posted earlier in the thread).

i'm not an nfs expert and have tried soft mounts, intr option but no luck. While the backup process fails I can successfully copy various files via command line to the nfs mounted location.

Suggestions welcome

 

[danny0n]

Did you find any solution for this problem ?

I have the same problem on one of my servers. What is weird is that I have a second cpanel server with the same configuration, same NFS server and everything works fine there.

Thank you.

 

[danny0n]

The problem seems to be with the horde database.

More specifically the process dies at line 1103 of pkgacct script

         eval { $horde->backup( $user, $horde_dir, $is_incremental ) };

For the moment I can't find that horde script.
If someone can point me to the script I could investigate this further.

Thank you.

 

[cPanelMichael]

The problem seems to be with the horde database.

More specifically the process dies at line 1103 of pkgacct script

         eval { $horde->backup( $user, $horde_dir, $is_incremental ) };

For the moment I can't find that horde script.
If someone can point me to the script I could investigate this further.

Can you backup the horde database manually? EX:

mysqldump horde > /path/to/horde_backup.sql

Thank you.

 

[danny0n]

Yes, mysqldump works as expected.
It would have been very weird if it didn't.

This problem only occurs when using a NFS for backups.
So probably is something weird happening in the horde backup script that kills the process and I don't think is database related. It works when there is no horde database probably because it skips everything.

 

[cPanelMichael]

Feel free to open a support ticket if you believe this is an issue with cPanel/WHM and not with your NFS mount. You can post the ticket number here so we can update this thread with the outcome.

Thank you.