Peformance Issue (gzipping on a NFS partition)

[phoenixweb]

Backup Performance Issue (gzipping on a NFS partition)

Hello everybody,

i'm experimenting a performance issue on my Backup Server due to the way cpanel makes backup.

THE LANDSCAPE:

• On my network I have a local backup server which I use to store backups of all my cpanel server (about 20).
• The speed of the internal Network is 1Gb LAN ethernet on a dedicated netcard.
• Every cPanel Server have a NFS mount on "/backup" which is a dir created on the backup server

WHAT CPANEL DO DURING A BACKUP:

1. Start making a copy of the account within "/backup/day/username/" which mean:
   -> SEND DATA TO THE BACKUP SERVER
2. gZip/pigZip the whole content of the directory created "/backup/day/username" which mean:
   <- READ IN STREAMING SINGLE FILE FROM THE BACKUP SERVER
   <-> PERFORM THE COMPRESSION DURING THE STREAMING
   -> WRITE THE FINAL GZIP FILE TO THE BACKUP SERVER

This affect the performance on the internal network smoothly in case of 1 server, but with 20 server you are gonna to kill the performance of every NFS mounts by filling up all the bandwith connections.

THE AWKARD TEMPORARY SOLUTION:
The awkard solution is to make the local backup on a different dir "/backup_tmp" and place the NFS backup dir on the alternate destination settings. Which means that i cannot use the new restore system AT ALL.

I think cPanel should make the whole "backup process" in a local tmp dirs and after move to the default local backup directory (which is the one used for restoring backup).

In order to fix there are two possibilities:
- use a tmp directory to backup till the gzip compression (avoid compression on a NFS network)
- or give the possibility to add local restore directory in order to see the backup on alternative destination


Thanks,
Massimiliano Cuttini

 

[phoenixweb]

Re: Backup Performance Issue (gzipping on a NFS partition)

My awkard solution doesn't work well as it will break up also the retention system.
I cannot delete old backups everyday of my life.

I think this is a real big problem.

 

[phoenixweb]

I have tried to open a Feature Request, but it was refused by telling me to use the same workaround I have done here.
The problem is that i'm gonna miss this feature by using an alternative destination:

1) Cannot use the restore system
2) Cannot retent/rotate log, i need to delete manually the files (everyday?)
3) System files are NOT moved but are leaved on the temp backup dir

Is there anybody from cPanel that can help me to solve this issue?
I understand that this problem can show only having many cpanel server, but the point is that cpanel should encorauge having many

Thanks,
Massimiliano Cuttini

 

[ThinIce]

It might be worth submitting a ticket so it's also logged from a support perspective, I can sort of understand why a feature request would be rejected on this, but having backup as an NFS mount isn't that unusual a use case from what I've seen in the past.

 

[phoenixweb]

Thanks ThinIce,

i have already submitted a support request, which is under investigation.
I'll post here updates if any

 

[cPanelMichael]

i have already submitted a support request, which is under investigation.
I'll post here updates if any

Could you let us know the ticket number so we can update this thread with the outcome?

Thank you.

 

[phoenixweb]

Hi Michael,

the support Request is Id 4276941 - Backup Performance Issue (gzipping on a NFS partition)
the internal case is 71461

The issue has been well explained by Gary Stanley:

I think when you gzip on a NFS partition the server is flooded with RPC calls, and high amounts of I/O wait which is killing your NFS server, if there was a way to compress them OFF the NFS server then copy them when they are compressed over to the server that will save a large amount of bandwidth/time.

 

[cPanelMichael]

There is a new case open to document the fact that the new backup system does all of its work under the default backup destination. For reference, the case number is 71749. While there are currently no immediate plans to change this behavior, I encourage you to open a feature request for this at:

Submit A Feature Request

You mentioned opening a feature request in the past. Could you let us know the specific link to that feature request?

Thank you.

 

[phoenixweb]

Everybody told me the same and I already did it BUT It had been refused!
Take a look below.

It's silly that everybody tell me to open a bug, open a feature request and cPanel refuse me both bug and feature request...... then? What's this for?

New Feature Request:

There is a performance issue with backups on a NFS mount.

WHAT CPANEL DO DURING A BACKUP:

Start making a copy of the account within "/backup/day/username/" which mean:

-> SEND DATA TO THE BACKUP SERVER

gZip/pigZip the whole content of the directory created "/backup/day/username" which mean:

<- READ IN STREAMING SINGLE FILE FROM THE BACKUP SERVER

<-> PERFORM THE COMPRESSION DURING THE STREAMING

-> WRITE THE FINAL GZIP FILE TO THE BACKUP SERVER

This way of make the backup waste a lot of network resource to save and retrive file from the NFS mount. Having many server that makes backup at the same time (nightly) means that the NFS network collapse and all the cpanel server will start to drammatically swapping, killing one by one all the other services.

It's really important change the way the backup is made by completing

the whole "backup process" for each account in a local tmp dirs, compress it locally and not above the NFS partition and ONLY AFTER move it to the default backup directory.

PRO:

- fastest backup

- no swapping if NFS network collapse

- no performance issue to slow disk writing / reading

Regards,

The cPanel Feature Requests Community Team

You are receiving this notification because you are an author of, or have posted a comment to, a Feature Request on the cPanel Feature Requests Community site! If you want to unsubscribe from this notification, please visit your profile page to update your settings for these types of emails.

 

[cPanelMichael]

I can't find that feature request at features.cpanel.net. Do you have the full URL for the request so we can review where it was declined?

Thank you.

 

[phoenixweb]

It had been declined by cPanel moderation.
As declined it was not admitted on feature portal.

I have only a simple email in my inbox from cPanel Feature:

Message-Id: <[email protected]>
From: cPanel Feature Requests
Reply-To: - removed -
Auto-Submitted: auto-generated
Precedence: bulk
Subject: This feature request was rejected. Reason: This capability already exists in the new backup system in 11.38. 1. Set your default backup directory to a chosen working directory2. Uncheck the box labeled "Retain backups in the default backup directory."3. Add an Additional Local Directory destination that corresponds to the mount point for your NFS mount.
To: - removed -
Date: Tue, 25 Jun 2013 12:00:28 +0000

 

[cPanelMichael]

Feel free to resubmit that feature request. I contacted our feature request moderators to let them know about this specific request and I have been assured it will be allowed through this time.

Thank you.