SOLVED Backup services and Eximstats Database

[Raymond Hayes]

This is a recent issue, and I have several other cpanel vps's without this problem.

When using any backup tool that is NOT the cpanel backup system, services fail. I get a message to this effect...

DBD::SQLite::db prepare failed: locking protocol at /usr/local/cpanel/scripts/eximstats_spam_check line 280.

After this happens, websites are unreachable or show blank pages, and the whole server seems to go into a lockdown. I am forced to reboot the server to get sites running again.

I know that external tools are not supported...but, at the same time, I use these tools on other VPSs with cpanel, and do not experience this issue.

Jetbackup and GK-Apps Restore Manager have both failed for backup solutions. So, it is also not a single service issue.

Thoughts on where I should start to look for solutions?

 

[cPanelLauren]

Hi @Raymond Hayes

That doesn't sound like the intended behavior at all considering you're using JetBackup which is supported. Is anything noted in the cPanel error logs or tailwatchd logs when this occurs?

/usr/local/cpanel/logs/error_log
/usr/local/cpanel/logs/tailwatchd_log

 

[Raymond Hayes]

Well, here is where the issue comes in. The noted errors are...

Cpanel::Exception::IO::FlockError/(XID tqt9be) The system failed to lock (LOCK_EX, LOCK_NB) the file “/var/cpanel/databases/vzrdsrxt.json” because of an error:
Cannot allocate memory

Several databases will display this error. But, backups will continue. Then when everything stops working, the log shows...

warn [cpanellogd] Cannot connect to database: /var/cpanel/eximstats_db.sqlite3: locking protocol (Cpanel::Exception::Database::Error
/(XID 2mbfex) The system received an error from the “SQLite” database “/var/cpanel/eximstats_db.sqlite3”: SQLITE_PROTOCOL (locking protocol)

Ok...so, I am running out of memory....maybe? That presents my next problem. Here are the stats with the VPS in question (currently...)

:free -mh
total used free shared buff/cache available
Mem: 3.0G 721M 983M 272M 1.3G 2.0G
Swap: 3.0G 610M 2.4G

This meets the required memory for Jetbackup (WHM Tweak Settings - Max cPanel process memory - require at least 2GB)
. So, it could be possible that it cranks up memory usage when the backup begins. I did try to enable "Conserve memory", but to no avail.

BUT, this presents a new issue. Here are the current stats on a different VPS where Jetbackup runs properly with no errors.

free -mh
total used free shared buff/cache available
Mem: 3.0G 1.2G 0B 1.5G 1.8G 1.5G
Swap: 3.0G 2.8G 202M

Jetbackup runs like clockwork every night...zero issues on this VPS. And this is a busy VPS.

Other ideas?

 

[cPanelLauren]

Well, it could be a memory issue. Seeing the memory usage right now may not be as useful opposed to when the issue is occurring. What filesystem are you using on this server and is it different than the one on your other one?

 

[Raymond Hayes]

Hello,

All are ext4. Basically duplicate machines.

As for memory usage, here is a sar -r during the backup sessions. I will separate the VPSs by the names Crash (forced to reboot, failing vps) and Non-crash (working vps). There is a discrepancy...

Crash
------------ kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
09:40:02 AM 0 3145728 100.00 0 2446516 0 0.00 895300 2061952 4484

Non-crash
------------ kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
12:40:01 AM 0 3145728 100.00 0 1941724 0 0.00 1685904 1110864 840

kbcached reflects more usage on the VPS that is crashing (2446516 crash vs 1941724 non-crashed) . BUT, as we move further into the numbers, it looks like a lot is being left unused on the crashed node

1. kbactive: 895300 Crash vs 1685904 Non-crash - The amount of active memory in kilobytes (memory that has been used more recently and usually not reclaimed unless absolutely necessary). - This shows the CRASH node having a lot less active memory, and should have more available to use...
2. kbinact: 2061952 Crash vs 1110864 Non-crash - The amount of inactive memory in kilobytes (memory which has been less recently used. It is more eligible to be reclaimed for other purposes) - This shows more memory being inactive on the CRASH node.

So, I then looked to swap usage.

Non-crash:
---------------kbswpfree kbswpused %swpused kbswpcad %swpcad
12:40:01 AM 190264 2955464 93.95 0 0.00

Crash:
---------------kbswpfree kbswpused %swpused kbswpcad %swpcad
09:40:02 AM 2543144 602584 19.16 0 0.00

(I checked vm.swappiness and both are set to 1, BTW)

So, you can see my dilemma here. It is like the CRASH VPS is refusing to use memory properly.

I know this starts to border on server administration a bit more...but, as well, when you have duplicate machines, with the less loaded unit crashing, while the heavily loaded vps does not...it brings questions with it.

 

[cPanelLauren]

Hi @Raymond Hayes

It does border a lot on System Administration but after seeing this I'd like to see if you would please open a ticket using the link in my signature? Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved.


Thanks!

 

[Raymond Hayes]

Hello,

I did want to follow up on this. The provider of the VPS has made changes to their VPS template. It limits usage for some items. Thus, memory is blocked for certain processes. We are moving to a new VPS to prevent this issue.

I appreciate your help and you may close this as resolved.

 

[cPanelLauren]

Hi @Raymond Hayes

I'm really glad you were able to find an answer for what was occurring and thank you for letting us know!