SOLVED Backup services and Eximstats Database

Raymond Hayes

Member
May 6, 2019
7
2
3
United States
cPanel Access Level
Root Administrator
This is a recent issue, and I have several other cpanel vps's without this problem.

When using any backup tool that is NOT the cpanel backup system, services fail. I get a message to this effect...

DBD::SQLite::db prepare failed: locking protocol at /usr/local/cpanel/scripts/eximstats_spam_check line 280.

After this happens, websites are unreachable or show blank pages, and the whole server seems to go into a lockdown. I am forced to reboot the server to get sites running again.

I know that external tools are not supported...but, at the same time, I use these tools on other VPSs with cpanel, and do not experience this issue.

Jetbackup and GK-Apps Restore Manager have both failed for backup solutions. So, it is also not a single service issue.

Thoughts on where I should start to look for solutions?
 

Raymond Hayes

Member
May 6, 2019
7
2
3
United States
cPanel Access Level
Root Administrator
Well, here is where the issue comes in. The noted errors are...

Cpanel::Exception::IO::FlockError/(XID tqt9be) The system failed to lock (LOCK_EX, LOCK_NB) the file “/var/cpanel/databases/vzrdsrxt.json” because of an error:
Cannot allocate memory


Several databases will display this error. But, backups will continue. Then when everything stops working, the log shows...

warn [cpanellogd] Cannot connect to database: /var/cpanel/eximstats_db.sqlite3: locking protocol (Cpanel::Exception::Database::Error
/(XID 2mbfex) The system received an error from the “SQLite” database “/var/cpanel/eximstats_db.sqlite3”: SQLITE_PROTOCOL (locking protocol)


Ok...so, I am running out of memory....maybe? That presents my next problem. Here are the stats with the VPS in question (currently...)

:free -mh
total used free shared buff/cache available
Mem: 3.0G 721M 983M 272M 1.3G 2.0G
Swap: 3.0G 610M 2.4G


This meets the required memory for Jetbackup (WHM Tweak Settings - Max cPanel process memory - require at least 2GB)
. So, it could be possible that it cranks up memory usage when the backup begins. I did try to enable "Conserve memory", but to no avail.

BUT, this presents a new issue. Here are the current stats on a different VPS where Jetbackup runs properly with no errors.

free -mh
total used free shared buff/cache available
Mem: 3.0G 1.2G 0B 1.5G 1.8G 1.5G
Swap: 3.0G 2.8G 202M

Jetbackup runs like clockwork every night...zero issues on this VPS. And this is a busy VPS.

Other ideas?
 

Raymond Hayes

Member
May 6, 2019
7
2
3
United States
cPanel Access Level
Root Administrator
Hello,

All are ext4. Basically duplicate machines.

As for memory usage, here is a sar -r during the backup sessions. I will separate the VPSs by the names Crash (forced to reboot, failing vps) and Non-crash (working vps). There is a discrepancy...

Crash
------------ kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
09:40:02 AM 0 3145728 100.00 0 2446516 0 0.00 895300 2061952 4484

Non-crash
------------ kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
12:40:01 AM 0 3145728 100.00 0 1941724 0 0.00 1685904 1110864 840

kbcached reflects more usage on the VPS that is crashing (2446516 crash vs 1941724 non-crashed) . BUT, as we move further into the numbers, it looks like a lot is being left unused on the crashed node
  1. kbactive: 895300 Crash vs 1685904 Non-crash - The amount of active memory in kilobytes (memory that has been used more recently and usually not reclaimed unless absolutely necessary). - This shows the CRASH node having a lot less active memory, and should have more available to use...
  2. kbinact: 2061952 Crash vs 1110864 Non-crash - The amount of inactive memory in kilobytes (memory which has been less recently used. It is more eligible to be reclaimed for other purposes) - This shows more memory being inactive on the CRASH node.
So, I then looked to swap usage.

Non-crash:
---------------kbswpfree kbswpused %swpused kbswpcad %swpcad
12:40:01 AM 190264 2955464 93.95 0 0.00

Crash:
---------------kbswpfree kbswpused %swpused kbswpcad %swpcad
09:40:02 AM 2543144 602584 19.16 0 0.00

(I checked vm.swappiness and both are set to 1, BTW)

So, you can see my dilemma here. It is like the CRASH VPS is refusing to use memory properly.

I know this starts to border on server administration a bit more...but, as well, when you have duplicate machines, with the less loaded unit crashing, while the heavily loaded vps does not...it brings questions with it.
 

Raymond Hayes

Member
May 6, 2019
7
2
3
United States
cPanel Access Level
Root Administrator
Hello,

I did want to follow up on this. The provider of the VPS has made changes to their VPS template. It limits usage for some items. Thus, memory is blocked for certain processes. We are moving to a new VPS to prevent this issue.

I appreciate your help and you may close this as resolved.
 
  • Like
Reactions: cPanelLauren