The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

RHEL5 Server Crash/Panic

Discussion in 'General Discussion' started by aag73, Mar 28, 2008.

  1. aag73

    aag73 Member

    Joined:
    Mar 28, 2008
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Mexico
    Starting March 26th, 2 different RHEL 5 servers have been crashing.

    Server 1 has crashed 1 time with Kernel 2.6.18-53.1.13
    Server 2 has crashed 1 time with Kernel 2.6.18-53.1.13PAE and 1 time with Kernel 2.6.18-53.1.14PAE

    Before the crashes, there are a few EXT3 messages which I include at the end, these EXT3 errors DO NOT show up on any logs going weeks back. These are brand new servers ordered in January and February, and they are distinct in data and hardware, so I don't think this is a hardware or data corruption issue.

    I have seen these type of EXT3 error messages on RHEL4 Kernel Crashes before. Typically going back to an older kernel solves the problem for a while, until Red Hat issues a new kernel that fixes the issue.

    I'll appreciate in someone could help me to debug the issue and share instructions on going back to an older RHEL 5 Kernel. I would do it myself, but the servers are so new they only have these newer kernels. I know about the local root exploit issue, but I don't issue SSH accounts to my customers, have compilers on only for root and my PHP config is pretty fortified, so that might not be as important.

    Many thanks!

    Details of errors:

    Server 1 Crash 1

    Mar 26 11:18:04 linux kernel: EXT3-fs warning (device sda8): dx_probe: Unrecognised inode hash code 56
    Mar 26 11:35:01 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9569664), 0
    Mar 26 12:18:40 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578081), 0
    Mar 26 12:20:21 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578051), 0
    Mar 26 12:20:21 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578073), 0
    Mar 26 12:24:35 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578089), 0
    Mar 26 12:24:35 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568695), 0
    Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568652), 0
    Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578053), 0
    Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9578054), 0
    Mar 26 12:42:06 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568690), 0
    Mar 26 12:54:53 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568703), 0
    Mar 26 12:55:03 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9577905), 0
    Mar 26 12:55:03 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (9568530), 0
    Mar 26 18:04:15 linux kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
    Mar 26 18:04:15 linux kernel: EXT3-fs: write access will be enabled during recovery.
    Mar 26 18:04:16 linux kernel: EXT3-fs: sda5: orphan cleanup on readonly fs
    Mar 26 18:04:16 linux kernel: EXT3-fs: sda5: 3 orphan inodes deleted
    Mar 26 18:04:16 linux kernel: EXT3-fs: recovery complete.
    Mar 26 18:04:16 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 26 18:04:17 linux kernel: EXT3 FS on sda5, internal journal
    Mar 26 18:04:17 linux kernel: EXT3 FS on sdb1, internal journal
    Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 26 18:04:17 linux kernel: EXT3 FS on sda1, internal journal
    Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 26 18:04:17 linux kernel: EXT3 FS on sda8, internal journal
    Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 26 18:04:17 linux kernel: EXT3 FS on sda6, internal journal
    Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 26 18:04:17 linux kernel: EXT3 FS on sda3, internal journal
    Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 26 18:04:17 linux kernel: EXT3 FS on sda2, internal journal
    Mar 26 18:04:17 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.

    Server 2 Crash 1

    Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233736), 0
    Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233765), 0
    Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233771), 0
    Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233748), 0
    Mar 27 10:17:07 linux kernel: EXT3-fs warning (device sda8): ext3_unlink: Deleting nonexistent file (21233749), 0
    Mar 27 10:32:49 linux kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
    Mar 27 10:32:49 linux kernel: EXT3-fs: write access will be enabled during recovery.
    Mar 27 10:32:49 linux kernel: EXT3-fs: sda5: orphan cleanup on readonly fs
    Mar 27 10:32:49 linux kernel: EXT3-fs: sda5: 1 orphan inode deleted
    Mar 27 10:32:49 linux kernel: EXT3-fs: recovery complete.
    Mar 27 10:32:49 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 27 10:32:50 linux kernel: EXT3 FS on sda5, internal journal
    Mar 27 10:32:50 linux kernel: EXT3 FS on sda8, internal journal
    Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 27 10:32:50 linux kernel: EXT3 FS on sdb1, internal journal
    Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 27 10:32:50 linux kernel: EXT3 FS on sda6, internal journal
    Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 27 10:32:50 linux kernel: EXT3 FS on sda3, internal journal
    Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 27 10:32:50 linux kernel: EXT3 FS on sda2, internal journal
    Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Mar 27 10:32:50 linux kernel: EXT3 FS on sda1, internal journal
    Mar 27 10:32:50 linux kernel: EXT3-fs: mounted filesystem with ordered data mode.


    Server 2 Crash 2

    Mar 28 01:10:32 linux kernel: EXT3-fs error (device sda8): htree_dirblock_to_tree: bad entry in directory #21233720: rec_len % 4 != 0 - offset=0, inode=1970562386, rec_len=28274, name_len=45
    Mar 28 01:10:32 linux kernel: EXT3-fs error (device sda8): ext3_journal_start_sb: Detected aborted journal
    Mar 28 01:10:33 linux kernel: EXT3-fs error (device sda8) in ext3_reserve_inode_write: Journal has aborted
    Mar 28 04:00:18 linux kernel: BUG: soft lockup detected on CPU#1!
    Mar 28 04:00:18 linux kernel: [<c044b2a4>] softlockup_tick+0x96/0xa4
    Mar 28 04:00:18 linux kernel: [<c042e214>] update_process_times+0x39/0x5c
    Mar 28 04:00:18 linux kernel: [<c04196ff>] smp_apic_timer_interrupt+0x5b/0x6c
    Mar 28 04:00:18 linux kernel: [<c04059bf>] apic_timer_interrupt+0x1f/0x24
    Mar 28 04:00:18 linux kernel: [<c0497061>] dqput+0x6b/0x15d
    Mar 28 04:00:18 linux kernel: [<c0498556>] vfs_quota_sync+0x9b/0x131
    Mar 28 04:00:18 linux kernel: [<c049a2dd>] quota_sync_sb+0x11/0xcc
    Mar 28 04:00:18 linux kernel: [<c0438d8d>] down_read+0x8/0x11
    Mar 28 04:00:18 linux kernel: [<c049a94c>] sys_quotactl+0x4c7/0x5f3
    Mar 28 04:00:18 linux kernel: [<c0464cc5>] free_pages_and_swap_cache+0x6b/0x7f
    Mar 28 04:00:18 linux kernel: [<c0605f6e>] do_page_fault+0x2a5/0x5d3
    Mar 28 04:00:18 linux kernel: [<c0605fe7>] do_page_fault+0x31e/0x5d3
    Mar 28 04:00:18 linux kernel: [<c0448c60>] audit_syscall_entry+0x11c/0x14e
    Mar 28 04:00:18 linux kernel: [<c0407ee7>] do_syscall_trace+0xab/0xb1
    Mar 28 04:00:18 linux kernel: [<c0404eff>] syscall_call+0x7/0xb
     
  2. minotauro

    minotauro Well-Known Member

    Joined:
    Jan 19, 2004
    Messages:
    89
    Likes Received:
    0
    Trophy Points:
    6
    Any solution to this issue? :confused:
     
  3. aag73

    aag73 Member

    Joined:
    Mar 28, 2008
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Mexico
    Are you having this problem as well? What version of kernel are you running?
     
  4. minotauro

    minotauro Well-Known Member

    Joined:
    Jan 19, 2004
    Messages:
    89
    Likes Received:
    0
    Trophy Points:
    6
    I´m having similiar issue. My server have RAID5 and I use the follow kernel:

    Linux 2.6.9-55.ELsmp #1 SMP Fri Apr 20 17:03:35 EDT 2007 i686 i686 i386 GNU/Linux

    The kernel panic always occuring when I try remove "lost" (unlinked) files (marked in red and user / group undefined - ?). Possible you have the same problem. When this unliked file is a mail, the server crash always when the user check the account. :mad:

    Your datacenter sugest any solution? Do you have this error too on /var/log/messages?

    Apr 23 10:14:42 s8 kernel: mm/memory.c:110: bad pmd e54ad110(f5a2414000000000).
    Apr 23 10:15:20 s8 kernel: mm/memory.c:110: bad pmd e54ad110(f5a2414000000000).
    Apr 23 10:15:57 s8 kernel: mm/memory.c:110: bad pmd e54ad110(f5a2414000000000).
    Apr 23 10:16:04 s8 kernel: mm/memory.c:110: bad pmd e54ad110(f5a2414000000000).
    Apr 23 10:16:07 s8 kernel: mm/memory.c:110: bad pmd e54ad110(f5a2414000000000).

    Regards,
    Minotauro.
     
Loading...

Share This Page