Update is blocked because RPM database is unstable

[carolainn]

Hello, I'm getting this error since a few weeks, and I don't know how to fix it, what should I do? It says that glibc is not installed, is that what I have to do? If so, where can I get it?

During a recent attempt to run /scripts/upcp, the system detected that your RPM database is unstable. The system cannot install any RPMs, so the upcp script cannot proceed.

FAIL: RPM DB error: rpmdb: Thread/process 3839/47267204490048 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 - (-30974)
error: cannot open Packages database in /var/lib/rpm
rpmdb: Thread/process 3839/47267204490048 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
package glibc is not installed
error: cannot open Packages database in /var/lib/rpm (exit code 1)

 

[cPanelLauren]

Hi @carolainn


Can you run the following via SSH and let me know if you continue to experience issues with the update:

/scripts/find_and_fix_rpm_issues

Alternatively, you can go to WHM>>Software>>Rebuild RPM Database to rebuild the database as well.

Thanks!

 

[4u123]

Since recent updates I've found a number of servers have started to have this problem every night. I can rebuild the RPM database from WHM or run the /scripts/find_and_fix_rpm_issues script - but 24 hours later the error is back. It seems each day a new server has this problem. These are all CL6 servers. I've not made any changes - the only common factor is recent cpanel updates.

My error is very slightly different to the one mentioned above - but same common problem "package glibc is not installed" -although I don't think it is missing.

FAIL: RPM DB error: rpmdb: Thread/process 619951/140267549230848 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 - (-30974)
error: cannot open Packages database in /var/lib/rpm
rpmdb: Thread/process 619951/140267549230848 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages database in /var/lib/rpm
package glibc is not installed (exit code 1)

 

[cPanelLauren]

Hi @4u123

That should definitely not be happening on a regular basis.
Can you please open a ticket using the link in my signature? Once open please reply with the Ticket ID here so that we can update this thread with the resolution once the ticket is resolved.


Thanks!

 

[kabatak]

Hi @cPanelLauren

I am also experiencing this issue right now, it's happen to me maybe 4-5 times in the last month.
I already have a ticket regarding this but the tech says it's not under Cpanel's care.
However like the OP, this seems to happen when there are Cpanel updates.
It keeps happening.

 

[4u123]

The solution for me was to run /scripts/find_and_fix_rpm_issues and then reboot the affected servers. Not sure what the problem was exactly but I've not had the issue on any of them since rebooting.

 

[kabatak]

@4u123

/scripts/find_and_fix_rpm_issues fixes the RPM issues but after a while, it will happen again.

After a few days, I will receive another "Update is blocked because RPM database is unstable" notification and the cycle keeps on repeating.

 

[4u123]

@4u123

/scripts/find_and_fix_rpm_issues fixes the RPM issues but after a while, it will happen again.

After a few days, I will receive another "Update is blocked because RPM database is unstable" notification and the cycle keeps on repeating.

Did you reboot the server as suggested in my previous reply?

 

[kabatak]

@4u123

I believe I've done that before but I'll try again.

 

[cPanelLauren]

Are your guys' servers running with high load/memory usage? I'm trying to think of circumstances when the RPM database would become unstable like this and it's typically when there's an issue with yum being able to run/complete. This wouldn't necessarily be cPanel related but will keep cPanel from being able to be updated.

 

[kabatak]

@cPanelLauren

My issue now appears to be related to CloudLinux license but I will need to observe for a few more weeks.

 

[cPanelLauren]

@cPanelLauren

My issue now appears to be related to CloudLinux license but I will need to observe for a few more weeks.

Hi @kabatak

That's interesting! What makes you believe it's related? I'd also like to know what you find out.

 

[kabatak]

@cPanelLauren

One of the techs pointed out that the CL license is not active so I activated it.

 

[cPanelLauren]

Hi @kabatak

Ahhh, that could cause some issues. Please let us know if it helps resolve the issue!

 

[4u123]

I wanted to provide an update to this because the problem is not resolved for me.

It seems every day at least one server has this issue. We never had a problem with this before 2018 when it started happening every day - then it stopped for a while and now it seems to be back - but not just on older busy servers. I have a number of new servers on CL7 with cpanel installed that are not fully in use yet - no clients on them - and still this problem happens on those servers too, as well as our existing servers that are in use.

Interestingly I noticed that /scripts/upcp does now seem to contain the find_and_fix_rpm_issues script to check the RPM database - but it doesn't actually work. I also noticed that when the rpm database is fine, you see this during upcp...

Processing: Checking RPM DB for corruption
[2019-10-11 09:26:11 +0100] - Processing command `/usr/local/cpanel/scripts/find_and_fix_rpm_issues`
[2019-10-11 09:26:13 +0100] [/usr/local/cpanel/scripts/find_and_fix_rpm_issues] Checking that the RPM DB is OK...
[2019-10-11 09:26:13 +0100] [/usr/local/cpanel/scripts/find_and_fix_rpm_issues] glibc-2.12-1.212.el6_10.3.x86_64
[2019-10-11 09:26:13 +0100] [/usr/local/cpanel/scripts/find_and_fix_rpm_issues] OK: RPM DB is responding to queries
[2019-10-11 09:26:13 +0100] [/usr/local/cpanel/scripts/find_and_fix_rpm_issues] Testing if rpm_is_working RPM is installed
[2019-10-11 09:26:13 +0100] [/usr/local/cpanel/scripts/find_and_fix_rpm_issues] package rpm_is_working is not installed
[2019-10-11 09:26:13 +0100] [/usr/local/cpanel/scripts/find_and_fix_rpm_issues] Testing if it's possible to install a simple RPM
[2019-10-11 09:26:13 +0100] [/usr/local/cpanel/scripts/find_and_fix_rpm_issues] Preparing... ##################################################
[2019-10-11 09:26:13 +0100] [/usr/local/cpanel/scripts/find_and_fix_rpm_issues] rpm_is_working ##################################################

However, when the rpm database is corrupt, the upcp script doesn't get this far before it bombs - so I think the running of this script within upcp needs moving up closer to the start. It would certainly save us the headache of running manually on at least one server every day.

I have no idea what the cause of this issue is. It seems completely random - but its clearly more of a Cloudlinux problem than a cpanel one.

 

[cPanelLauren]

Hi @4u123


If you have your CloudLinux license through us I'd suggest opening a ticket with us so we can look further into it. If you purchased it from somewhere else you can open a ticket with CloudLinux directly here: CloudLinux - Main | New template