Unlinked inode... Remounting filesystem read-only - AGAIN?

[deieno]

Hi...
I have this server that is becaming on filesystem read-only mode EVERY 2:00 (around).

logs show:


Jan 25 02:50:31 server pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
Jan 25 02:50:42 server pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__GjI_22MRgfyqZTv544N TA0j7feqymxLP0P1XXzWvvbWzZrNHfkR89BAtuEKbnNQP is now logged in
Jan 25 02:50:43 server pure-ftpd: (__cpanel__service__auth__ftpd__GjI_22MRgfyqZTv544 NTA0j7feqymxLP0P1XXzWvvbWzZrNHfkR89BAtuEKbnNQP@127 .0.0.1) [INFO] Logout.
Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222206 in dir #159220259
Jan 25 02:51:44 server kernel: Aborting journal on device sda8.
Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222215 in dir #159220259
Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222143 in dir #159220259
Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222125 in dir #159220259
Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222250 in dir #159220259
Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222235 in dir #159220259
Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222127 in dir #159220259
Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222210 in dir #159220259
Jan 25 02:51:44 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222150 in dir #159220259
Jan 25 02:51:45 server kernel: EXT3-fs error (device sda8): ext3_lookup: unlinked inode 159222236 in dir #159220259
Jan 25 02:51:45 server kernel: ext3_abort called.
Jan 25 02:51:45 server kernel: EXT3-fs error (device sda8): ext3_journal_start_sb: Detected aborted journal
Jan 25 02:51:45 server kernel: Remounting filesystem read-only


root@server [/var/log]# grep only messages
Jan 25 02:51:45 server kernel: Remounting filesystem read-only
Jan 25 07:56:19 server kernel: Write protecting the kernel read-only data: 392k
Jan 25 07:56:21 server kernel: EXT3-fs: INFO: recovery required on readonly filesystem.



Ok.. I scheduled a FSCK... but the day after that ... same issue.

What worries me is not the problem itself... but, last week this happened in TWO servers... One at Softlayer.. one at theplanet... Yes.. I thought it was to much coincidence... but, "things" happens..

The solution for the last week case was a OS RELOAD and THAN... restoring everything ... well.. A nightmare.. all night awake restoring the data.

I thought my nighs were safe.. at least for more time...
But now, another client server with the SAME issue.

Strangely.. it turn just the /home partition on read-only mode.. the way it happened with the other servers that had the os reload.

Before start to trying things... I would like to have some opinions here. What this could be?

A cpanel issue? a centos issue? someone trying to hurt the server?
or it's really just happens and the fact it happen twice on other servers last week it's just coincidence?

I appreciate any help...

Thank you

[madaboutlinux]

This looks like a kernel problem. What kernel are you using on this server and was it the same kernel on the other 2 servers when the problem occurred last week?

You can find the kernel by executing the command:

uname -a

[deieno]

Hi...
kernel is
Linux hostname 2.6.18-92.el5PAE #1 SMP Tue Jun 10 19:22:41 EDT 2008 i686 i686 i386 GNU/Linux


thank you

[zigzam]

I have run into this exact same issue. Happens one time daily at roughly 2 am. I am using kernel 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:32:21 EST 2010 x86_64 x86_64 x86_64 GNU/Linux


Any ideas?

[rackaid]

Would these systems happen to have Adaptec RAId controllers?

I've seen this issue if the RAID never fully initializes.


I've also seen issues with ext3 that can be fixed by downgrading to ext2. Run a fsck. Then go back to ext3. Re-run fsck.

Lastly, perhaps you do have bad blocks in an area and something at that time happens to hit that area.

[zigzam]

Mine is a Adaptec 2405

Everything is reported as optimal.