kernel panic...

[4u123]

Hi,

Are there any super brains out there who can help me decipher what happened here ?

Server was using 2.6.9-42.0.8.ELsmp

Mar 5 01:49:53 hs006 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000114
Mar 5 01:49:53 hs006 kernel: printing eip:
Mar 5 01:49:53 hs006 kernel: c016f651
Mar 5 01:49:53 hs006 kernel: *pde = 03bb6001
Mar 5 01:49:53 hs006 kernel: Oops: 0000 [#1]
Mar 5 01:49:53 hs006 kernel: SMP
Mar 5 01:49:53 hs006 kernel: Modules linked in: loop ipt_owner iptable_mangle ip_conntrack_ftp ipt_conntrack ipt_REJECT ipt_LOG ipt_limit ipt_multiport ipt_state ip_conntrack iptable_filter ip_tables md5 ipv6 autofs4 button battery ac uhci_hcd ehci_hcd e1000 dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod ata_piix libata sd_mod scsi_mod
Mar 5 01:49:53 hs006 kernel: CPU: 1
Mar 5 01:49:53 hs006 kernel: EIP: 0060:[<c016f651>] Not tainted VLI
Mar 5 01:49:53 hs006 kernel: EFLAGS: 00010206 (2.6.9-42.0.8.ELsmp)
Mar 5 01:49:53 hs006 kernel: EIP is at prune_dcache+0x290/0x31b
Mar 5 01:49:53 hs006 kernel: eax: 00000100 ebx: c21fb840 ecx: e9a10500 edx: c0393000
Mar 5 01:49:53 hs006 kernel: esi: d933af24 edi: d933af2c ebp: e9a105c8 esp: f7cfdee8
Mar 5 01:49:53 hs006 kernel: ds: 007b es: 007b ss: 0068
Mar 5 01:49:53 hs006 kernel: Process kswapd0 (pid: 57, threadinfo=f7cfd000 task=f7d276f0)
Mar 5 01:49:53 hs006 kernel: Stack: 00000000 00000009 00000000 00000083 00000000 f7ffe9c0 c016fa24 c01497f0
Mar 5 01:49:53 hs006 kernel: 00250300 00000000 00000004 00000000 000773a0 000000d0 00000020 c032b780
Mar 5 01:49:53 hs006 kernel: 00000002 c0328f80 00000002 c014aa8b c02d27c9 000773a0 f7cfdf9c 00000020
Mar 5 01:49:53 hs006 kernel: Call Trace:
Mar 5 01:49:53 hs006 kernel: [<c016fa24>] shrink_dcache_memory+0x16/0x2d
Mar 5 01:49:53 hs006 kernel: [<c01497f0>] shrink_slab+0xf8/0x161
Mar 5 01:49:53 hs006 kernel: [<c014aa8b>] balance_pgdat+0x1e1/0x30e
Mar 5 01:49:53 hs006 kernel: [<c02d27c9>] schedule+0x86d/0x8db
Mar 5 01:49:53 hs006 kernel: [<c0120420>] prepare_to_wait+0x12/0x4c
Mar 5 01:49:53 hs006 kernel: [<c014ac82>] kswapd+0xca/0xcc
Mar 5 01:49:53 hs006 kernel: [<c01204f5>] autoremove_wake_function+0x0/0x2d
Mar 5 01:49:53 hs006 kernel: [<c02d47f2>] ret_from_fork+0x6/0x14
Mar 5 01:49:53 hs006 kernel: [<c01204f5>] autoremove_wake_function+0x0/0x2d
Mar 5 01:49:53 hs006 kernel: [<c014abb8>] kswapd+0x0/0xcc
Mar 5 01:49:53 hs006 kernel: [<c01041f5>] kernel_thread_helper+0x5/0xb
Mar 5 01:49:53 hs006 kernel: Code: 46 40 8b 56 40 8b 48 04 89 4a 04 89 11 89 40 04 89 46 40 89 f8 e8 7b 3e 16 00 b8 00 30 39 c0 e8 71 3e 16 00 8b 46 4c 85 c0 74 0f <8b> 48 14 85 c9 74 08 89 ea 89 f0 ff d1 eb 1a 89 e8 e8 12 24 00
Mar 5 01:49:53 hs006 kernel: <0>Fatal exception: panic in 5 seconds

[chirpy]

Probably the significant part of that is Process kswapd0 (pid: 57, threadinfo=f7cfd000 task=f7d276f0) which is the process that swaps out memory. It could be a memory chip problem, but it more likely a kernel issue. I would suggest upgrading to 2.6.9-42.0.10.ELsmp which has some stability fixes in it.

[AndyReed]

Are there any super brains out there who can help me decipher what happened here ?

Server was using 2.6.9-42.0.8.ELsmp

Although the problem might be the kernel, make sure the RAM Memory is in a good working condition. Ask your host to test your RAM. Update to the latest Kernel v2.6.9-42.0.10.ELsmp, but if that doesn't help, you have to either manually re-compile the Kernel, or do OS reload (this should be your last resrot.)

[4u123]

Thanks for the advice, I upgraded to the latest kernel then the server crashed again...

Mar 6 01:29:35 hs006 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000114
Mar 6 01:29:35 hs006 kernel: printing eip:
Mar 6 01:29:35 hs006 kernel: c016f675
Mar 6 01:29:35 hs006 kernel: *pde = 0ac70001
Mar 6 01:29:35 hs006 kernel: Oops: 0000 [#1]
Mar 6 01:29:35 hs006 kernel: SMP
Mar 6 01:29:35 hs006 kernel: Modules linked in: loop ipt_owner iptable_mangle ip_conntrack_ftp ipt_conntrack ipt_REJECT ipt_LOG ipt_limit ipt_multiport ipt_state ip_conntrack iptable_filter ip_tables md5 ipv6 autofs4 button battery ac uhci_hcd ehci_hcd e1000 dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod ata_piix libata sd_mod scsi_mod
Mar 6 01:29:35 hs006 kernel: CPU: 1
Mar 6 01:29:35 hs006 kernel: EIP: 0060:[<c016f675>] Not tainted VLI
Mar 6 01:29:35 hs006 kernel: EFLAGS: 00010206 (2.6.9-42.0.10.ELsmp)
Mar 6 01:29:35 hs006 kernel: EIP is at prune_dcache+0x290/0x31b
Mar 6 01:29:35 hs006 kernel: eax: 00000100 ebx: c21fbc40 ecx: f5a17900 edx: c0393000
Mar 6 01:29:35 hs006 kernel: esi: d933af24 edi: d933af2c ebp: f5a179d8 esp: f7cfdee8
Mar 6 01:29:35 hs006 kernel: ds: 007b es: 007b ss: 0068
Mar 6 01:29:35 hs006 kernel: Process kswapd0 (pid: 57, threadinfo=f7cfd000 task=f7d276f0)
Mar 6 01:29:35 hs006 kernel: Stack: 00000000 00000026 00000000 00000081 00000000 f7ffe9c0 c016fa48 c0149814
Mar 6 01:29:35 hs006 kernel: 001d1a00 00000000 00000003 00000000 00077fbc 000000d0 00000020 c032a380
Mar 6 01:29:35 hs006 kernel: 00000001 c0328f80 00000002 c014aaaf c02d27f5 00077fbc f7cfdf9c 00000000
Mar 6 01:29:35 hs006 kernel: Call Trace:
Mar 6 01:29:35 hs006 kernel: [<c016fa48>] shrink_dcache_memory+0x16/0x2d
Mar 6 01:29:35 hs006 kernel: [<c0149814>] shrink_slab+0xf8/0x161
Mar 6 01:29:35 hs006 kernel: [<c014aaaf>] balance_pgdat+0x1e1/0x30e
Mar 6 01:29:35 hs006 kernel: [<c02d27f5>] schedule+0x86d/0x8db
Mar 6 01:29:36 hs006 kernel: [<c0120420>] prepare_to_wait+0x12/0x4c
Mar 6 01:29:36 hs006 kernel: [<c014aca6>] kswapd+0xca/0xcc
Mar 6 01:29:36 hs006 kernel: [<c01204f5>] autoremove_wake_function+0x0/0x2d
Mar 6 01:29:36 hs006 kernel: [<c02d481e>] ret_from_fork+0x6/0x14
Mar 6 01:29:36 hs006 kernel: [<c01204f5>] autoremove_wake_function+0x0/0x2d
Mar 6 01:29:36 hs006 kernel: [<c014abdc>] kswapd+0x0/0xcc
Mar 6 01:29:36 hs006 kernel: [<c01041f5>] kernel_thread_helper+0x5/0xb
Mar 6 01:29:36 hs006 kernel: Code: 46 40 8b 56 40 8b 48 04 89 4a 04 89 11 89 40 04 89 46 40 89 f8 e8 83 3e 16 00 b8 00 30 39 c0 e8 79 3e 16 00 8b 46 4c 85 c0 74 0f <8b> 48 14 85 c9 74 08 89 ea 89 f0 ff d1 eb 1a 89 e8 e8 1e 24 00
Mar 6 01:29:36 hs006 kernel: <0>Fatal exception: panic in 5 seconds
Mar 6 09:38:36 hs006 syslogd 1.4.1: restart.


I'll swap out the memory with new modules - hopefully that will fix it. Thanks for the advice.