Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

SOLVED Backup services and Eximstats Database

Discussion in 'Database Discussion' started by Raymond Hayes, May 6, 2019.

  1. Raymond Hayes

    Raymond Hayes Registered

    Joined:
    May 6, 2019
    Messages:
    4
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    United States
    cPanel Access Level:
    Root Administrator
    This is a recent issue, and I have several other cpanel vps's without this problem.

    When using any backup tool that is NOT the cpanel backup system, services fail. I get a message to this effect...

    DBD::SQLite::db prepare failed: locking protocol at /usr/local/cpanel/scripts/eximstats_spam_check line 280.

    After this happens, websites are unreachable or show blank pages, and the whole server seems to go into a lockdown. I am forced to reboot the server to get sites running again.

    I know that external tools are not supported...but, at the same time, I use these tools on other VPSs with cpanel, and do not experience this issue.

    Jetbackup and GK-Apps Restore Manager have both failed for backup solutions. So, it is also not a single service issue.

    Thoughts on where I should start to look for solutions?
     
  2. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    6,210
    Likes Received:
    478
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @Raymond Hayes

    That doesn't sound like the intended behavior at all considering you're using JetBackup which is supported. Is anything noted in the cPanel error logs or tailwatchd logs when this occurs?

    /usr/local/cpanel/logs/error_log
    /usr/local/cpanel/logs/tailwatchd_log
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Raymond Hayes

    Raymond Hayes Registered

    Joined:
    May 6, 2019
    Messages:
    4
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    United States
    cPanel Access Level:
    Root Administrator
    Well, here is where the issue comes in. The noted errors are...

    Cpanel::Exception::IO::FlockError/(XID tqt9be) The system failed to lock (LOCK_EX, LOCK_NB) the file “/var/cpanel/databases/vzrdsrxt.json” because of an error:
    Cannot allocate memory


    Several databases will display this error. But, backups will continue. Then when everything stops working, the log shows...

    warn [cpanellogd] Cannot connect to database: /var/cpanel/eximstats_db.sqlite3: locking protocol (Cpanel::Exception::Database::Error
    /(XID 2mbfex) The system received an error from the “SQLite” database “/var/cpanel/eximstats_db.sqlite3”: SQLITE_PROTOCOL (locking protocol)


    Ok...so, I am running out of memory....maybe? That presents my next problem. Here are the stats with the VPS in question (currently...)

    :free -mh
    total used free shared buff/cache available
    Mem: 3.0G 721M 983M 272M 1.3G 2.0G
    Swap: 3.0G 610M 2.4G


    This meets the required memory for Jetbackup (WHM Tweak Settings - Max cPanel process memory - require at least 2GB)
    . So, it could be possible that it cranks up memory usage when the backup begins. I did try to enable "Conserve memory", but to no avail.

    BUT, this presents a new issue. Here are the current stats on a different VPS where Jetbackup runs properly with no errors.

    free -mh
    total used free shared buff/cache available
    Mem: 3.0G 1.2G 0B 1.5G 1.8G 1.5G
    Swap: 3.0G 2.8G 202M

    Jetbackup runs like clockwork every night...zero issues on this VPS. And this is a busy VPS.

    Other ideas?
     
  4. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    6,210
    Likes Received:
    478
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Well, it could be a memory issue. Seeing the memory usage right now may not be as useful opposed to when the issue is occurring. What filesystem are you using on this server and is it different than the one on your other one?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. Raymond Hayes

    Raymond Hayes Registered

    Joined:
    May 6, 2019
    Messages:
    4
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    United States
    cPanel Access Level:
    Root Administrator
    Hello,

    All are ext4. Basically duplicate machines.

    As for memory usage, here is a sar -r during the backup sessions. I will separate the VPSs by the names Crash (forced to reboot, failing vps) and Non-crash (working vps). There is a discrepancy...

    Crash
    ------------ kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
    09:40:02 AM 0 3145728 100.00 0 2446516 0 0.00 895300 2061952 4484

    Non-crash
    ------------ kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
    12:40:01 AM 0 3145728 100.00 0 1941724 0 0.00 1685904 1110864 840

    kbcached reflects more usage on the VPS that is crashing (2446516 crash vs 1941724 non-crashed) . BUT, as we move further into the numbers, it looks like a lot is being left unused on the crashed node
    1. kbactive: 895300 Crash vs 1685904 Non-crash - The amount of active memory in kilobytes (memory that has been used more recently and usually not reclaimed unless absolutely necessary). - This shows the CRASH node having a lot less active memory, and should have more available to use...
    2. kbinact: 2061952 Crash vs 1110864 Non-crash - The amount of inactive memory in kilobytes (memory which has been less recently used. It is more eligible to be reclaimed for other purposes) - This shows more memory being inactive on the CRASH node.
    So, I then looked to swap usage.

    Non-crash:
    ---------------kbswpfree kbswpused %swpused kbswpcad %swpcad
    12:40:01 AM 190264 2955464 93.95 0 0.00

    Crash:
    ---------------kbswpfree kbswpused %swpused kbswpcad %swpcad
    09:40:02 AM 2543144 602584 19.16 0 0.00

    (I checked vm.swappiness and both are set to 1, BTW)

    So, you can see my dilemma here. It is like the CRASH VPS is refusing to use memory properly.

    I know this starts to border on server administration a bit more...but, as well, when you have duplicate machines, with the less loaded unit crashing, while the heavily loaded vps does not...it brings questions with it.
     
  6. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    6,210
    Likes Received:
    478
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @Raymond Hayes

    It does border a lot on System Administration but after seeing this I'd like to see if you would please open a ticket using the link in my signature? Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved.


    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. Raymond Hayes

    Raymond Hayes Registered

    Joined:
    May 6, 2019
    Messages:
    4
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    United States
    cPanel Access Level:
    Root Administrator
    Hello,

    I did want to follow up on this. The provider of the VPS has made changes to their VPS template. It limits usage for some items. Thus, memory is blocked for certain processes. We are moving to a new VPS to prevent this issue.

    I appreciate your help and you may close this as resolved.
     
    cPanelLauren likes this.
  8. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    6,210
    Likes Received:
    478
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @Raymond Hayes

    I'm really glad you were able to find an answer for what was occurring and thank you for letting us know!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice