Backups taking longer on CloudLinux and cPanel 66

[feanorknd]

Hello:

just have actually same experience.... backups are taking almost 18-20 hours..... are incremental, only keeping actual day, at NFS partition.

Before last upgrade to cpanel v64, it was taking maybe 2-3 hours or less.

Any information related???

Thanks.

 

[cPanelMichael]

just have actually same experience.... backups are taking almost 18-20 hours..... are incremental, only keeping actual day, at NFS partition.

Before last upgrade to cpanel v64, it was taking maybe 2-3 hours or less.

Any information related???

There are a couple of cases referenced on the following post:

Remote incremental backups - timeouts

However, as far as the speed itself, could you open a support ticket using the link in my signature so we can take a closer look?

Thank you.

 

[mindnetcombr]

Hello

Im having the same issue here.

Let me tell my scenario: previous server running CL6 64bits, cpanel 11.66. Incremental backup took 2 or 3 hours to complete, every night. Low CPU load.

So we moved all accounts to a new server, better CPU, double RAM, running CloudLinux 7 64 bits, cpanel 11.66. Now incremental backups need from 8 to 12 hours to complete. When the backup start at 01am, load rise from 0.XX to 2.00.

Something is wrong, because it's same accounts (in fact less accounts because some users leave the new server), more CPU, more memory, new hard drives. We try replace the backup hard drive, but this dont fix the issue. How the same backup job took 3 hours on a old server and 8 to 12 hours on a more powerfull server?

The main change beside the hardware is CL6 to CL7.

(fully updated cpanel, kernel)

Thank

 

[mindnetcombr]

Just a update:

I use this small script to update the incremental backup:

KONTA=`ls -1A /var/cpanel/users/`
for x in `echo -n "$KONTA"`;do
if [ -d "/home/$x" ]; then
/usr/local/cpanel/bin/pkgacct --incremental "$x" /backup/cpbackup/daily backup
fi
done

Result? backup done in less than 3 hours - like on previous server.

If I run incremental backup using the legacy backup OR the new backup system, the result is the same: 12 hours to complete backup.

Something is throttling the backup script (legacy and new) of cpanel, I dont know if is CloudLinux 7 related or anything else.

 

[InterServed]

Same problems for me. I had to split accounts into more cpanel servers just to make the backups complete in reasonable time.
I believe it may have to do with the new meta files and/or because the changes cpanel made in the way incrementals are made/processed. Pruning of the old backup dir after the backup finishes as well , in my case just the pruning sometimes took 1-2 hours or more). So i personally never saw any backup improvements ever since v60 , it only degraded more in terms of performance/impact on the systems.

One more thing that I'm not sure if it is related or not , but on systems with MariaDB 10.2 i notice very high memory increase , probably memory leaks.

I will probably start testing the method mindnetcombr described , until cpanel will decide to remove that as well.

 

[mindnetcombr]

One more thing that I'm not sure if it is related or not , but on systems with MariaDB 10.2 i notice very high memory increase , probably memory leaks.

I will probably start testing the method mindnetcombr described , until cpanel will decide to remove that as well.

I dont use MariaDB, so I think is not related.

I try use legacy backup system (incremental) and same issue. When I use a simple script to pkgacct all accounts (the script on my previous message above), server have no issues.

The more strange for me is: this start only when I move from CL6 to CL7. I have servers running CL6 doing incremental backups daily in less than 3 hours (650 accounts average, 1Tb of data average).

Sure meta files on new backup system is problem too - and like you said, pruning the old days cause high load on my servers.

Based on my experience the new backup system is a total disaster.

I hope they keep the script pkgacct - so I can run my own backup script, without any issues.

 

[cPanelMichael]

Hello,

Please feel free to open a support ticket if you are facing this issue so we can take a closer look. Post the ticket number here and we will update this thread with the outcome.

Thank you.

 

[mindnetcombr]

Update: now I have 2 servers on same condition, cPanel + CloudLinux 7, incremental backups take 12 hours.

If I run the script below, backup complete in 3 hours:

KONTA=`ls -1A /var/cpanel/users/`
for x in `echo -n "$KONTA"`;do
if [ -d "/home/$x" ]; then
/usr/local/cpanel/bin/pkgacct --incremental "$x" /backup/cpbackup/daily backup
fi
done

 

[cPanelMichael]

Hi @mindnetcombr,

Could you open a support ticket using the link in my signature so we can take a closer look at one of the affected systems? You can post the ticket number here and we will update this thread with the outcome.

Thank you.

 

[mindnetcombr]

Hi @mindnetcombr,

Could you open a support ticket using the link in my signature so we can take a closer look at one of the affected systems? You can post the ticket number here and we will update this thread with the outcome.

Thank you.

Hi

I have the ticket #8858129 handled by cPanel Shoshana.

Your support team is looking on my two servers at same time, doing some research and I think they found a possibly culprit for this, but I'll wait for the investigation to finish before I say anything. You can read on ticket.

Thank Michael!

 

[cPanelMichael]

Hello,

We are tracking reports of backups taking significantly longer to complete (compared to cPanel version 64) on servers using cPanel version 66 and CloudLinux as part of internal case CPANEL-15825. I'll monitor this case and update this thread with more information as it becomes available.

Thank you.

 

[mindnetcombr]

Hello,

I have two servers running CloudLinux 7 and cPanel 11.66, in version 11.64 it took 3 hours, now in version 11.66 each backup takes 12 hours.

This has turned into a daily hell, because backups take so long that they eventually reach business hours, and the server gets overloaded.

I know that the developers are researching, it was me who opened the ticket that culminated in the opening of the investigation case they are doing, but the days are passing and there is no solution. I have this problem for 15 days.

Is there any way to downgrade from 11.66 to 11.64? I believe the 11.64 version dont suffer this problem, If it were possible I might have an immediate solution.

 

[cPanelMichael]

Hello,

Our developers are currently in communication with CloudLinux to further investigate the cause of this issue. I'll continue to monitor the case and update this thread with more information as it becomes available.

Is there any way to downgrade from 11.66 to 11.64? I believe the 11.64 version dont suffer this problem, If it were possible I might have an immediate solution.

You can not downgrade to an earlier major version (e.g. 66 to 64) after updating.

Thank you.

 

[mindnetcombr]

More info maybe usefull for someone else: CloudLinux 6 dont have this issue. I have servers running CloudLinux 6 and backup run as usual. But all my servers with CloudLinux 7 have the same isse.

So if you have a server with CloudLinux 6 dont upgrade to 7 until this issue is fixed.

 

[zye]

my backup took also 11 hours today but i just noticed that it takes every 7 days that long... ( 1 TB of data )

 

[mindnetcombr]

Hello cPanelMichael,

Any progress on this issue?

Thank

 

[cPanelMichael]

Hi @mindnetcombr,

There's no update to report at this time. I'll continue to monitor the case and update this thread once new information is available.

Thank you.

 

[InterServed]

Hello,

Backups run perfect using the latest v68 build and activating the "Disable phpMyAdmin information schema searches" which can be found in the Tweak settings. This is much better. Will monitor more servers in the next days.

backups start at 00:01

v66
-rw------- 1 root root 882K Oct 12 03:50 1507755722.log
-rw------- 1 root root 880K Oct 13 03:26 1507842122.log
-rw------- 1 root root 884K Oct 14 04:16 1507928521.log
-rw------- 1 root root 882K Oct 15 03:36 1508014922.log
-rw------- 1 root root 884K Oct 16 04:01 1508101322.log
-rw------- 1 root root 885K Oct 17 04:03 1508187722.log
-rw------- 1 root root 889K Oct 18 04:33 1508274122.log
-rw------- 1 root root 877K Oct 19 03:38 1508360522.log
-rw------- 1 root root 869K Oct 20 04:22 1508446921.log
-rw------- 1 root root 869K Oct 21 02:49 1508533321.log
v68
-rw------- 1 root root 882K Oct 22 01:42 1508619722.log

 

[mindnetcombr]

Hello,

Backups run perfect using the latest v68 build and activating the "Disable phpMyAdmin information schema searches" which can be found in the Tweak settings. This is much better. Will monitor more servers in the next days.

Hello InterServed,

Thank you for the information. How are going the tests with cpanel 11.68? Are you using CloudLinux 6 or 7?

Here I found this: if I reboot the server (CloudLinux 7) and run backup as soon server boot, the incremental backups run quick.

But after 12-24 hours of a boot, the issue begins.

 

[mindnetcombr]

Hello,

Backups run perfect using the latest v68 build and activating the "Disable phpMyAdmin information schema searches" which can be found in the Tweak settings. This is much better. Will monitor more servers in the next days.

Hello, InterServed

Yesterday I update to 11.68.09 (release) CloudLinux 7 and this not fixed the issue.

Unfortunately