cpbackup and memory issue

[hostmach]

We have been experiencing out of memory issues on WHM v96 when cpbackup tries to execute a first backup run on a new server with 1.3TB of account data with the following specs:

OS: Cloudlinux 7
CPU: Intel(R) Xeon(R) E-2136 CPU @ 3.30GHz
RAM: 64GB ram
Storage: Main drive is SSD and Backup drive is SSD.

We have other servers running v94 with inferior specs that never face this issue during a first backup run (first backup run as in if the backup drive fails and we need to replace it and run a first set all over again):

OS: Cloudlinux 7
CPU: Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz
RAM: 16GB ram
Storage: Main drive is SSD and Backup drive is SATA.

When this OOM issue happens, load average goes above 100 and server starts swapping like crazy. It gets into an unrecoverable point where the only solution is to reboot the server.

We tried to perform backups on 3 different occasions which all ended up crashing the server (forcing us to reboot from an unrecoverable state).

We have been in touch with cPanel Support which wants us to submit the server (and our customers) to another backup attempt so they can monitor what is going on as cpbackup runs but we can't afford to subject our customers to more downtime due to this issue.

Is anybody else experiencing OOM issues with cpbackup on v96?

 

[cPRex]

Hey there! I haven't heard of anyone else having a similar issue. Is there possible a hardware problem on that server, indicated by high I/O wait times?

What happens if you run a manual pkgacct on the domain itself with this command?

/scripts/pkgacct username

You can stop that process on the command line at any time with a Ctrl + C if it does start to cause load.

 

[hostmach]

Hello,

"Is there possible a hardware problem on that server, indicated by high I/O wait times?"

The 2 drives are both SSD drives. I/O wait times are fine.

What happens if you run a manual pkgacct on the domain itself with this command?

Maybe I used the wrong wording when I said "run on a new server with 1.3TB of account data", maybe you got the wrong idea this is a single 1.3TB account server?

No, this isn't a single account server, there are 700 accounts on this server (just like any other server we have with inferior hardware specs) hence the 1.3TB.

Running pkacct on a single account works fine, the backup runs fine for the first 2 hours and then memory suddenly spikes and the server get into this unrecoverable state.

 

[kodeslogic]

What are the values set for the below-listed settings in WHM >> Home >> Server Configuration >> Tweak Settings

Extra CPUs for server load
Critical load threshold
gzip compression level
Number of pigz processes

 

[hostmach]

What are the values set for the below-listed settings in WHM >> Home >> Server Configuration >> Tweak Settings

Extra CPUs for server load
Critical load threshold
gzip compression level
Number of pigz processes

The same we use for our other servers with inferior hardware:

Extra CPUs for server load: default (0
Critical load threshold: 8
gzip compression level: default (6)
Number of pigz processes: default (1)

 

[cPRex]

@hostmach - you're always welcome to submit a ticket to our team so we can check the issue directly on the affected system. I don't have any other similar complaints at this time, so we might need to take a look.