SOLVED Corrupted directories - /usr/local/bin and apache

[billmcollam]

I recently noticed a couple of error messages from CPANEL CRON

Cron <[email protected]> /usr/bin/test -x /usr/local/cpanel/scripts/update_db_cache && /usr/local/cpanel/scripts/update_db_cache

stat(/usr/local/bin/psql): Input/output error
stat(/usr/local/bin/postmaster): Input/output error

When I looked into /usr/local/ i find two corrupted directories (bin and apache).

[[email protected] usr]# cd local
[[email protected] local]# ls -la
ls: cannot access bin: Input/output error
ls: cannot access apache: Input/output error
total 56
drwxr-xr-x 16 root root  4096 Jun 18 00:40 .
drwxr-xr-x 13 root root  4096 Apr  1 22:07 ..
d?????????  ? ?    ?        ?            ? apache
d?????????  ? ?    ?        ?            ? bin
drwx--x--x 39 root wheel 4096 Jul  3 00:41 cpanel
drw-------  6 root root  4096 Jun 30  2020 csf
drwxr-xr-x  2 root root  4096 Nov  5  2016 etc
drwxr-xr-x  2 root root  4096 Nov  5  2016 games
drwxr-xr-x  7 root root  4096 Jun 30  2020 include
drwxr-xr-x  3 root root  4096 Jun 30  2020 lib
drwxr-xr-x  3 root root  4096 Jun 30  2020 lib64
drwxr-xr-x  2 root root  4096 Nov  5  2016 libexec
drwxr-xr-x  3 root root  4096 Jun 30  2020 man
drwxr-xr-x  2 root root  4096 Jun 30  2020 sbin
drwxr-xr-x  7 root root  4096 Jun 30  2020 share
drwxr-xr-x  2 root root  4096 Nov  5  2016 src

The site seems to be otherwise working, although I'm having trouble keeping rsyslogd up and running (perhaps related)?

Anyone with any idea how the directories got corrupted and how to repair/replace them?

Thanks in advance.

 

[andrew.n]

Is it a VPS or a dedicated server? input/output errors are usually occurs due to disk corruptions and the disk might be failing. I would suggest to run an FSCK check on those disks to ensure the filesystem is okay.

 

[billmcollam]

THanks for the response. It's a VPS ... I ran tune2fs and it shows filestate as "clean with errors", then

FS Error count:           819937
First error time:         Sat Apr  3 03:06:15 2021
First error function:     ext4_iget
First error line #:       4199
First error inode #:      1972045
First error block #:      0
Last error time:          Sun Jul  4 09:09:34 2021
Last error function:      ext4_iget
Last error line #:        4199
Last error inode #:       1713738
Last error block #:       0

For now I've sent this to my hosting company. I guess the next step is to force fsck (I'm just unsure where to look for log results after rebooting).
Any other advice is welcomed.

Cheers

 

[andrew.n]

If this is a VPS then yeah your hosting provider should be able to assist you.

 

[billmcollam]

Yep.. they ran fsck in recovery, which cleared all the fs errors.
The corrupted directorys were removed. I then ran WHM upgrade CPANEL which recovered the directorys.
Happy campers nows.
Thanks a million,

 

[andrew.n]

lovely to hear, enjoy!

 

[cPRex]

Thanks for the advice, @andrew.n. When you see question marks in the file output, that's an indicator of some level of disk or file system corruption. I'm glad the host was able to help get that resolved!