[CPANEL-14896] Restoring Incremental Backups from NSFv4 mounts

[dzung042]

After upgrade I have error restore account,
"Could not restore account “uyozcizu”: Account Restore Failed: “The system could not load the cPanel user attributes file from the archive because of an error: (XID 98vm2v) The system failed to lock (LOCK_EX, LOCK_NB) the file “/backup/2017-09-18/accounts/cpanelpkgrestore.TMP.work.O3eDqQLI/safe_to_read_archive/contents/cp/uyozcizu” because of an error: Bad file descriptor at /usr/local/cpanel/Whostmgr/Transfers/Utils.pm line 234."
Any one help me resolve

 

[cPanelMichael]

Hello,

Can you verify if the location of your backup directory is configured as a NFS mount?

Thank you.

 

[feday2]

Hello,

Can you verify if the location of your backup directory is configured as a NFS mount?

Thank you.

I have the same problem, and /backup/ - NFS mount disk.

 

[cPanelMichael]

Hello,

Internal case CPANEL-14896 is open to report an issue where it's not possible to restore incremental backups from a NFSv4 mount. I'll monitor this case and update this thread with the outcome. In the meantime, a temporary workaround is to move the incremental backup directory that you want to restore outside of the NFS mount and restore it using a command like this:

/scripts/restorepkg /path/to/inremental/user/

One additional workaround to consider is to set the NFS mount options to "local_lock=all,nolock" to see if that helps.

Thank you.

 

[Intercloud]

Hi cPanelMichael,
Even though I am using this forum for years, I have just created my first account solely to answer you.

I am having the exact same problem with about 3 of my WHM servers (could be more, but these are the ones I found out about).

The scenario in my case is a version 4 NFS server that is mounted to each of my WHM servers, and is used to save their backups that are created using WHM intergrated backup system. I am not using incremental backups but only full backups of all cPanel accounts.

I also get this kind of message when trying to restore a backup:
Could not restore account “rotemintercloud”: Account Restore Failed: “The system could not load the cPanel user attributes file from the archive because of an error: (XID k8mqzx) The system failed to lock (LOCK_EX, LOCK_NB) the file “/mnt/backup/2017-09-19/accounts/cpanelpkgrestore.TMP.work.lCXCWZG5/safe_to_read_archive/contents/cp/rotemintercloud” because of an error: Bad file descriptor at /usr/local/cpanel/Whostmgr/Transfers/Utils.pm line 234.
”

I have also noticed that the WHM backup system's retention is no longer working and old backups are simply not being deleted.

This issue has started about a month ago, following one of WHM's updates I believe. Seems like a bug to me since it worked perfectly in this exact way for years now (I have over 40 WHM servers that are configured in the exact same way).

About your two temporary workaround:

1. Copying the backup file localy and then running it using a script / "Restore a Full Backup/cpmove File" WHM tool is working perfectly, but it is not a solution since it is very time-consuming and my customers won't know how to do it themselves (They restore backups using WHM's web interface).

2. Setting the NFS mount options to "local_lock=all,nolock" doesn't seem to be working on my case.

3. Changing the mount's NFS version to 3 instead of 4 seem to be solving the problem completely, but I can only implement it to WHM servers that are mounted to one of my NFS servers that is supporting NFSv3 (I cannot change the other NFS server version since it is not supporting it).


In conclusion, it seems to be a bug that have just recently started and should be fixed ASAP since it hurts backup retentions and restores.

Thanks a lot for your attention and assistance on this matter.

 

[cPanelMichael]

Hello @Intercloud,

I've linked this forums thread to internal case CPANEL-14896. I'll continue to monitor this case and update this thread with more information as soon as it's available. Please also note the following FAQ entry:

Backup Configuration FAQ - Version 66 Documentation - cPanel Documentation

Thank you.

 

[orudge]

We are also experiencing this issue, which only started in the past month or so. We have backup servers mounted via NFS, and our clients are unable to restore backups due to a "The system failed to lock (LOCK_EX, LOCK_NB)" error. Altering the backup mount in fstab to the suggested line unfortunately does not help. Nor does using nfsvers=3.

Perhaps the cPanel code could be altered to store the temporarily extracted data in a different location (e.g., /home)?

Thanks,

 

[cPanelMichael]

Hello,

There's no update to report on the status of internal case CPANEL-14896 at this time. I'll continue to monitor the case and update this thread with more information as it becomes available. As far as a workaround, you should be able to move the backup archives to a non-NFS mount location and then proceed with the restore.

Thank you.

 

[Intercloud]

Hi cPanelMichael,
Are there any news? Is there an estimated time for the fix to be released?

The thing is that we have plenty of customer's WHM servers that are currently having many backup and restore problems.
We are working with NFS based backups for our WHM servers for years now without any problems, and if this issue will take much to be solved we will need to start planning on finding an alternative solution.
This will take a lot of time and efforts which we really prefer to avoid at the moment, so a rough estimation of the period of time for this issue to be fix will be really helpful.

Thanks a lot for your assistance!

 

[cPanelMichael]

Hello,

It looks like we've completed work on a resolution, and it's now in under testing. There's currently no specific time frame for the publication of the resolution, but I'll monitor the case and update this thread as soon as more information is available.

Thank you.

 

[orudge]

Hi Michael, just wondering if you have any further update on this case? I've not seen anything in any of the changelogs about it.

Thanks,

 

[cPanelMichael]

Hi Michael, just wondering if you have any further update on this case? I've not seen anything in any of the changelogs about it.

Hi,

Internal case CPANEL-14896 is still open, however we did publish a resolution for another NFS related issue that may help alleviate the issue on some systems:

Fixed case CPANEL-16122: Improve cpmove restore compatibility with NFS.

If the issue is still occurring, you are welcome to open a support ticket so we can take a closer look at your system and link your specific report to internal case CPANEL-14896.

Thank you.