[Case 74517] Legacy Backup - missing emails

[uk01]

Hi Everyone,

We are slowly migrating accounts to our new servers and had 5 instances in the last month where clients have called us saying emails have disappeared from their IMAP folders. The folders are there, but no emails.

Tonight, we got hit again, with an account we migrated last week (yes it took a week for them to tell us, by which time we'd deleted the account from the old server!)

5 hours later... I've possible BUG bug in cpanel.

Our old server is still set on Legacy backups and the transfer/migrate function uses the same backup generator. On inspecting the backup files on our backup server we found them to be corrupted.

We tried opening them in Mac OSX expander, it comes up with an error. An older version of "Stuffit" opened the archive but files were missing.
Eventually we tried the latest version of stuffit and found the missing files. For some odd reason, the backup archive has a second "home" folder inside.

username/home
aswell as
username/username/homedir

The extra home folder contains the missing files. When cpanel restores from the backup during a transfer, it doesn't restore these files, so they appear missing.

I checked some very old backups, prior to when CPanel made the legacy / new backup system and everything is ok. The problem has occurred since it stopped compressing the home directory inside the main archive.


To restore the missing files, I copied them directly to the users account then chown username.username -R /home/username/mail to update the ownership.

Check your backups! If you ever need them, they may be corrupt. We have now switched the old server to the new backup system.***UPDATE BELOW: Problem remains with new backups***

Hope this saves you some pain!

 

[uk01]

I've just discovered that the new server puts the lost files in a directory called "home" - so after transfering accounts, check this folder and move any files into the relevant accounts, then chown ownership.

 

[cPanelMichael]

Hello

I could not reproduce the behavior you have reported on a test server. All email data was backed up and restored successfully using /scripts/pkgacct and the transfer feature. Could you open a support ticket if you are able to reproduce the issue with emails not backing up successfully? You can submit a ticket via:

Submit A Ticket

Please post the ticket number here so we can update this thread with the outcome.

Thank you.

 

[uk01]

Hi there,

It took me 9 hours last night to restore all the files "orphaned" into the /home/home folder into the correct user accounts home/user/ on the new server. All the orphaned files were IMAP folders and sent items. It was random too, for example it only affected some folders and some emails.

For example, a user had 5 imap folders and 1500 sent items. It would orphan 3 of the folders and 800 sent items, which I had to manually move back into the users mailbox.

We switched all remaining accounts on the old server to the new backup system last night. This morning downloaded one of the problem archives and the error is still in the archive - see image below



Here you can see the top folder is the proper user folder containing all files for account restore then under it is a second stray home folder, this contains the orphaned files.
During account restore or migration, these orphaned files get thrown into a /home/home/ folder.

Tonight I'm going to run our backups fro the old web server to a different backup server to see if the problem continues.
I'm also going to check the backups of the accounts I restored last night.

This might narrow down where the issue is and if it's only on our old server or more widespread.
After this clarification, I will file a ticket with you for investigation.

Cheers

 

[cPanelMichael]

Remember to post the ticket number here after submitting a ticket so we can update this thread.

Thank you.

 

[uk01]

This is to confirm that the issue remains with our new backup folder. It also remains, even with the migrated accounts where missing emails have been restored (from the orphaned home folder).

When these accounts are backed up, backups are corrupt and the backup system moves various emails/files into a stray home folder. If these backups were used for disaster recovery they would not be reliable and would not restore the account fully without manually moving the files into their correct location.

We pay for cpanel, we expect backups (which are extremely important) to be reliable. I will now be logging a support ticket and will post the ticket number here.

 

[uk01]

Support Request Id 4308941 - Corrupt backups

 

[cPanelMichael]

It was determined this issue was related to a flaw in the Archive::Tar::Builder Perl module. For reference, the internal case number is 74517. A resolution is scheduled for inclusion with cPanel version 11.38.3 and 11.40 (11.39 development build). The change log can be monitored for the case number:

cPanel - Change Log

Thank you.

 

[uk01]

I would like to highlight the excellent support CPanel have offered on this issue - it was a hard one to solve, needing many hours of intricate problem solving both ends!

**Special mention for Richard Asp - who never let it go, 2 nights running until he found the bug**

The support guys stuck with it and eventually narrowed down the issue - this is the first time I've had any dealings with Cpanel support in several years of hosting and I was amazed how helpful they are.
Thanks guys, I look forward to the fix coming to the release build and once again can sleep at night knowing my backups are 100%