aag73

Member
Mar 28, 2008
20
0
51
Mexico
Starting March 26th, 2 different RHEL 5 servers have been crashing.

Server 1 has crashed 1 time with Kernel 2.6.18-53.1.13
Server 2 has crashed 1 time with Kernel 2.6.18-53.1.13PAE and 1 time with Kernel 2.6.18-53.1.14PAE

Before the crashes, there are a few EXT3 messages which I include at the end, these EXT3 errors DO NOT show up on any logs going weeks back. These are brand new servers ordered in January and February, and they are distinct in data and hardware, so I don't think this is a hardware or data corruption issue.

I have seen these type of EXT3 error messages on RHEL4 Kernel Crashes before. Typically going back to an older kernel solves the problem for a while, until Red Hat issues a new kernel that fixes the issue.

I'll appreciate in someone could help me to debug the issue and share instructions on going back to an older RHEL 5 Kernel. I would do it myself, but the servers are so new they only have these newer kernels. I know about the local root exploit issue, but I don't issue SSH accounts to my customers, have compilers on only for root and my PHP config is pretty fortified, so that might not be as important.

Many thanks!

Details of errors:

Server 1 Crash 1

Mar 26 11:18:04 linux kernel: EXT3-fs warning (device sda8): dx_probe: Unrecognised inode hash code 56
Mar 26 11:35:01 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9569664), 0
Mar 26 12:18:40 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578081), 0
Mar 26 12:20:21 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578051), 0
Mar 26 12:20:21 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578073), 0
Mar 26 12:24:35 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578089), 0
Mar 26 12:24:35 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568695), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568652), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578053), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578054), 0
Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568690), 0
Mar 26 12:54:53 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568703), 0
Mar 26 12:55:03 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9577905), 0
Mar 26 12:55:03 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568530), 0
Mar 26 18:04:15 linux kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
Mar 26 18:04:15 linux kernel: EXT3-fs: write access will be enabled during recovery.
Mar 26 18:04:16 linux kernel: EXT3-fs: sda5: orphan cleanup on readonly fs
Mar 26 18:04:16 linux kernel: EXT3-fs: sda5: 3 orphan inodes deleted
Mar 26 18:04:16 linux kernel: EXT3-fs: recovery complete.
Mar 26 18:04:16 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda5, internal journal
Mar 26 18:04:17 linux kernel: EXT3 FS on sdb1, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda1, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda8, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda6, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda3, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 26 18:04:17 linux kernel: EXT3 FS on sda2, internal journal
Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.

Server 2 Crash 1

Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233736), 0
Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233765), 0
Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233771), 0
Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233748), 0
Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233749), 0
Mar 27 10:32:49 linux kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
Mar 27 10:32:49 linux kernel: EXT3-fs: write access will be enabled during recovery.
Mar 27 10:32:49 linux kernel: EXT3-fs: sda5: orphan cleanup on readonly fs
Mar 27 10:32:49 linux kernel: EXT3-fs: sda5: 1 orphan inode deleted
Mar 27 10:32:49 linux kernel: EXT3-fs: recovery complete.
Mar 27 10:32:49 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sda5, internal journal
Mar 27 10:32:50 linux kernel: EXT3 FS on sda8, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sdb1, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sda6, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sda3, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sda2, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
Mar 27 10:32:50 linux kernel: EXT3 FS on sda1, internal journal
Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.


Server 2 Crash 2

Mar 28 01:10:32 linux kernel: EXT3-fs error (device sda8): htree_dirblock_to_tree: bad entry in directory #21233720: rec_len % 4 != 0 - offset=0, inode=1970562386, rec_len=28274, name_len=45
Mar 28 01:10:32 linux kernel: EXT3-fs error (device sda8): ext3_journal_start_sb: Detected aborted journal
Mar 28 01:10:33 linux kernel: EXT3-fs error (device sda8) in ext3_reserve_inode_write: Journal has aborted
Mar 28 04:00:18 linux kernel: BUG: soft lockup detected on CPU#1!
Mar 28 04:00:18 linux kernel: [<c044b2a4>] softlockup_tick+0x96/0xa4
Mar 28 04:00:18 linux kernel: [<c042e214>] update_process_times+0x39/0x5c
Mar 28 04:00:18 linux kernel: [<c04196ff>] smp_apic_timer_interrupt+0x5b/0x6c
Mar 28 04:00:18 linux kernel: [<c04059bf>] apic_timer_interrupt+0x1f/0x24
Mar 28 04:00:18 linux kernel: [<c0497061>] dqput+0x6b/0x15d
Mar 28 04:00:18 linux kernel: [<c0498556>] vfs_quota_sync+0x9b/0x131
Mar 28 04:00:18 linux kernel: [<c049a2dd>] quota_sync_sb+0x11/0xcc
Mar 28 04:00:18 linux kernel: [<c0438d8d>] down_read+0x8/0x11
Mar 28 04:00:18 linux kernel: [<c049a94c>] sys_quotactl+0x4c7/0x5f3
Mar 28 04:00:18 linux kernel: [<c0464cc5>] free_pages_and_swap_cache+0x6b/0x7f
Mar 28 04:00:18 linux kernel: [<c0605f6e>] do_page_fault+0x2a5/0x5d3
Mar 28 04:00:18 linux kernel: [<c0605fe7>] do_page_fault+0x31e/0x5d3
Mar 28 04:00:18 linux kernel: [<c0448c60>] audit_syscall_entry+0x11c/0x14e
Mar 28 04:00:18 linux kernel: [<c0407ee7>] do_syscall_trace+0xab/0xb1
Mar 28 04:00:18 linux kernel: [<c0404eff>] syscall_call+0x7/0xb
 

minotauro

Well-Known Member
Jan 19, 2004
89
0
156
Are you having this problem as well? What version of kernel are you running?
I´m having similiar issue. My server have RAID5 and I use the follow kernel:

Linux 2.6.9-55.ELsmp #1 SMP Fri Apr 20 17:03:35 EDT 2007 i686 i686 i386 GNU/Linux

The kernel panic always occuring when I try remove "lost" (unlinked) files (marked in red and user / group undefined - ?). Possible you have the same problem. When this unliked file is a mail, the server crash always when the user check the account. :mad:

Your datacenter sugest any solution? Do you have this error too on /var/log/messages?

Apr 23 10:14:42 s8 kernel: mm/memory.c:110: bad pmd e54ad110(f5a2414000000000).
Apr 23 10:15:20 s8 kernel: mm/memory.c:110: bad pmd e54ad110(f5a2414000000000).
Apr 23 10:15:57 s8 kernel: mm/memory.c:110: bad pmd e54ad110(f5a2414000000000).
Apr 23 10:16:04 s8 kernel: mm/memory.c:110: bad pmd e54ad110(f5a2414000000000).
Apr 23 10:16:07 s8 kernel: mm/memory.c:110: bad pmd e54ad110(f5a2414000000000).

Regards,
Minotauro.