SOLVED Corrupted directories - /usr/local/bin and apache

billmcollam

Member
Jan 12, 2015
22
2
53
Ontario Canada
cPanel Access Level
Reseller Owner
I recently noticed a couple of error messages from CPANEL CRON
Code:
Cron <[email protected]> /usr/bin/test -x /usr/local/cpanel/scripts/update_db_cache && /usr/local/cpanel/scripts/update_db_cache

stat(/usr/local/bin/psql): Input/output error
stat(/usr/local/bin/postmaster): Input/output error
When I looked into /usr/local/ i find two corrupted directories (bin and apache).

Code:
[[email protected] usr]# cd local
[[email protected] local]# ls -la
ls: cannot access bin: Input/output error
ls: cannot access apache: Input/output error
total 56
drwxr-xr-x 16 root root  4096 Jun 18 00:40 .
drwxr-xr-x 13 root root  4096 Apr  1 22:07 ..
d?????????  ? ?    ?        ?            ? apache
d?????????  ? ?    ?        ?            ? bin
drwx--x--x 39 root wheel 4096 Jul  3 00:41 cpanel
drw-------  6 root root  4096 Jun 30  2020 csf
drwxr-xr-x  2 root root  4096 Nov  5  2016 etc
drwxr-xr-x  2 root root  4096 Nov  5  2016 games
drwxr-xr-x  7 root root  4096 Jun 30  2020 include
drwxr-xr-x  3 root root  4096 Jun 30  2020 lib
drwxr-xr-x  3 root root  4096 Jun 30  2020 lib64
drwxr-xr-x  2 root root  4096 Nov  5  2016 libexec
drwxr-xr-x  3 root root  4096 Jun 30  2020 man
drwxr-xr-x  2 root root  4096 Jun 30  2020 sbin
drwxr-xr-x  7 root root  4096 Jun 30  2020 share
drwxr-xr-x  2 root root  4096 Nov  5  2016 src
The site seems to be otherwise working, although I'm having trouble keeping rsyslogd up and running (perhaps related)?

Anyone with any idea how the directories got corrupted and how to repair/replace them?

Thanks in advance.
 
Last edited by a moderator:

andrew.n

Well-Known Member
Jun 9, 2020
633
183
43
EU
cPanel Access Level
Root Administrator
Is it a VPS or a dedicated server? input/output errors are usually occurs due to disk corruptions and the disk might be failing. I would suggest to run an FSCK check on those disks to ensure the filesystem is okay.
 

billmcollam

Member
Jan 12, 2015
22
2
53
Ontario Canada
cPanel Access Level
Reseller Owner
THanks for the response. It's a VPS ... I ran tune2fs and it shows filestate as "clean with errors", then
Code:
FS Error count:           819937
First error time:         Sat Apr  3 03:06:15 2021
First error function:     ext4_iget
First error line #:       4199
First error inode #:      1972045
First error block #:      0
Last error time:          Sun Jul  4 09:09:34 2021
Last error function:      ext4_iget
Last error line #:        4199
Last error inode #:       1713738
Last error block #:       0
For now I've sent this to my hosting company. I guess the next step is to force fsck (I'm just unsure where to look for log results after rebooting).
Any other advice is welcomed.

Cheers
 

andrew.n

Well-Known Member
Jun 9, 2020
633
183
43
EU
cPanel Access Level
Root Administrator
If this is a VPS then yeah your hosting provider should be able to assist you.
 

billmcollam

Member
Jan 12, 2015
22
2
53
Ontario Canada
cPanel Access Level
Reseller Owner
Yep.. they ran fsck in recovery, which cleared all the fs errors.
The corrupted directorys were removed. I then ran WHM upgrade CPANEL which recovered the directorys.
Happy campers nows.
Thanks a million,