The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server crash - can you decipher this for me ?

Discussion in 'General Discussion' started by 4u123, Aug 14, 2007.

  1. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    Can anyone translate this debug information and give me a clue as to what may have caused the server to crash ?

    Aug 14 02:09:44 server kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000100
    Aug 14 02:09:44 server kernel: printing eip:
    Aug 14 02:09:44 server kernel: f88a016b
    Aug 14 02:09:44 server kernel: *pde = 2245e001
    Aug 14 02:09:44 server kernel: Oops: 0000 [#1]
    Aug 14 02:09:44 server kernel: SMP
    Aug 14 02:09:44 server kernel: Modules linked in: ipt_owner iptable_mangle ip_conntrack_ftp ipt_conntrack ipt_REJECT ipt_LOG ipt_limit ipt_multiport ipt_state ip_conntrack iptable_filter ip_tables md5 ipv6 autofs4 loop button battery ac uhci_hcd ehci_hcd e1000 dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod ata_piix libata sd_mod scsi_mod
    Aug 14 02:09:44 server kernel: CPU: 1
    Aug 14 02:09:44 server kernel: EIP: 0060:[<f88a016b>] Not tainted VLI
    Aug 14 02:09:44 server kernel: EFLAGS: 00010206 (2.6.9-55.0.2.ELsmp)
    Aug 14 02:09:44 server kernel: EIP is at __journal_remove_checkpoint+0xb/0x65 [jbd]
    Aug 14 02:09:44 server kernel: eax: f2571ecc ebx: 00000100 ecx: f2571ecc edx: eef4d4c0
    Aug 14 02:09:44 server kernel: esi: eef4d4c0 edi: f7e99400 ebp: eef4d4c0 esp: f7cfce0c
    Aug 14 02:09:44 server kernel: ds: 007b es: 007b ss: 0068
    Aug 14 02:09:44 server kernel: Process kswapd0 (pid: 57, threadinfo=f7cfc000 task=f7d276f0)
    Aug 14 02:09:44 server kernel: Stack: f2571ecc eef4d4c0 f889d973 eef4d4c0 c1f26460 f2571ecc f889da19 00000000
    Aug 14 02:09:44 server kernel: f7e99400 f88664d3 000000d0 f42fe1e8 f7cfcf58 c015d5c7 f42fe1e8 c1f26460
    Aug 14 02:09:44 server kernel: c0149e93 00000001 00000001 00000001 00000000 f7cfcec8 f7cfce64 f7cfce64
    Aug 14 02:09:44 server kernel: Call Trace:
    Aug 14 02:09:44 server kernel: [<f889d973>] __journal_try_to_free_buffer+0x69/0x89 [jbd]
    Aug 14 02:09:44 server kernel: [<f889da19>] journal_try_to_free_buffers+0x86/0xc5 [jbd]
    Aug 14 02:09:44 server kernel: [<f88664d3>] ext3_releasepage+0x0/0x54 [ext3]
    Aug 14 02:09:44 server kernel: [<c015d5c7>] try_to_release_page+0x34/0x46
    Aug 14 02:09:44 server kernel: [<c0149e93>] shrink_list+0x293/0x3ed
    Aug 14 02:09:44 server kernel: [<c0149090>] __pagevec_release+0x15/0x1d
    Aug 14 02:09:44 server kernel: [<c014a1ca>] shrink_cache+0x1dd/0x34d
    Aug 14 02:09:44 server kernel: [<c014a888>] shrink_zone+0xa7/0xb6
    Aug 14 02:09:44 server kernel: [<c014ace7>] balance_pgdat+0x1c5/0x30e
    Aug 14 02:09:44 server kernel: [<c02d3f36>] schedule+0x87e/0x8ec
    Aug 14 02:09:44 server kernel: [<c0120458>] prepare_to_wait+0x12/0x4c
    Aug 14 02:09:44 server kernel: [<c014aefa>] kswapd+0xca/0xcc
    Aug 14 02:09:44 server kernel: [<c012052d>] autoremove_wake_function+0x0/0x2d
    Aug 14 02:09:44 server kernel: [<c02d5fae>] ret_from_fork+0x6/0x14
    Aug 14 02:09:44 server kernel: [<c012052d>] autoremove_wake_function+0x0/0x2d
    Aug 14 02:09:44 server kernel: [<c014ae30>] kswapd+0x0/0xcc
    Aug 14 02:09:44 server kernel: [<c01041f5>] kernel_thread_helper+0x5/0xb
    Aug 14 02:09:44 server kernel: Code: 0a 89 d8 e8 30 f9 ff ff 01 04 24 39 eb 75 e0 3b 74 24 08 75 c3 8b 04 24 83 c4 0c 5b 5e 5f 5d c3 56 89 c1 53 8b 58 24 85 db 74 57 <8b> 33 c7 40 24 00 00 00 00 8b 51 2c 8b 40 28 89 50 2c 8b 51 2c
    Aug 14 02:09:44 server kernel: <0>Fatal exception: panic in 5 seconds
    Aug 14 02:09:45 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:30:48:89:c9:d2:00:0b:46:cb:81:00:08:00 SRC=217.132.94.151 DST=89.21.3.90 LEN=52 TOS=0x00 PREC=0x00 TTL=55 ID=52250 DF PROTO=TCP SPT=34343 DPT=2967 WINDOW=60352 RES=0x00 SYN URGP=0
     
  2. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    Happened again - same server. Lots of FTP activity - then the server crashes. I'd appreciate any help anyone could give me here.

    Aug 19 22:17:31 server pure-ftpd: (user@x.x.x.x) [NOTICE] /home/user//public_html/reports.htm uploaded (252626 bytes, 43.73KB/sec)
    Aug 19 22:17:38 server pure-ftpd: (user@x.x.x.x) [NOTICE] /home/user//public_html/reports0506.htm uploaded (331720 bytes, 44.44KB/sec)
    Aug 19 22:17:46 server pure-ftpd: (user@x.x.x.x) [NOTICE] /home/user//public_html/reports0607.htm uploaded (310954 bytes, 43.36KB/sec)
    Aug 19 22:17:46 server pure-ftpd: (user@x.x.x.x) [NOTICE] /home/user//public_html/reports0708.htm uploaded (21398 bytes, 45.44KB/sec)
    Aug 19 22:17:47 server pure-ftpd: (user@x.x.x.x) [NOTICE] /home/user//public_html/stats.htm uploaded (52821 bytes, 45.27KB/sec)
    Aug 19 22:17:49 server pure-ftpd: (user@x.x.x.x) [NOTICE] /home/user//public_html/stats0405.htm uploaded (64658 bytes, 45.16KB/sec)
    Aug 19 22:17:51 server pure-ftpd: (user@x.x.x.x) [NOTICE] /home/user//public_html/stats0506.htm uploaded (94584 bytes, 45.22KB/sec)
    Aug 19 22:17:53 server pure-ftpd: (user@x.x.x.x) [NOTICE] /home/user//public_html/stats0607.htm uploaded (77076 bytes, 45.23KB/sec)
    Aug 19 22:20:16 server pure-ftpd: (user@x.x.x.x) [NOTICE] /home/user//public_html/gotyoursmall.jpg uploaded (4487 bytes, 47.82KB/sec)
    Aug 19 22:20:17 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:30:48:89:c9:d2:00:0b:46:cb:81:00:08:00 SRC=203.171.236.7 DST=89.21.3.93 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=30168 DF PROTO=TCP SPT=39634 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0
    Aug 19 22:20:17 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:30:48:89:c9:d2:00:0b:46:cb:81:00:08:00 SRC=203.171.236.7 DST=89.21.3.92 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=1390 DF PROTO=TCP SPT=39633 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0
    Aug 19 22:20:17 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:30:48:89:c9:d2:00:0d:ed:18:03:80:08:00 SRC=203.171.236.7 DST=x.x.x.x LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=5295 DF PROTO=TCP SPT=39631 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0
    Aug 19 22:20:17 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:30:48:89:c9:d2:00:0d:ed:18:03:80:08:00 SRC=203.171.236.7 DST=89.21.3.94 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=22408 DF PROTO=TCP SPT=39635 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0
    Aug 19 22:20:17 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:30:48:89:c9:d2:00:0b:46:cb:81:00:08:00 SRC=203.171.236.7 DST=89.21.3.91 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=1899 DF PROTO=TCP SPT=39632 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0
    Aug 19 22:20:20 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:30:48:89:c9:d2:00:0b:46:cb:81:00:08:00 SRC=203.171.236.7 DST=89.21.3.93 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=30170 DF PROTO=TCP SPT=39634 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0
    Aug 19 22:20:45 server pure-ftpd: (user@x.x.x.x) [NOTICE] /home/user//public_html/index.htm uploaded (11846 bytes, 45.61KB/sec)
    Aug 19 22:21:16 server pure-ftpd: (?@x.x.x.x) [INFO] New connection from x.x.x.x
    Aug 19 22:21:16 server pure-ftpd: (?@x.x.x.x) [INFO] user is now logged in
    Aug 19 22:21:16 server pure-ftpd: (user@x.x.x.x) [INFO] Can't change directory to /www/_mm: No such file or directory
    Aug 19 22:21:16 server pure-ftpd: (user@x.x.x.x) [INFO] Can't change directory to /www/_mm/: No such file or directory
    Aug 19 22:21:17 server pure-ftpd: (user@x.x.x.x) [INFO] Can't change directory to /www/_mm: No such file or directory
    Aug 19 22:21:17 server pure-ftpd: (user@x.x.x.x) [INFO] Can't change directory to /www/_mm/: No such file or directory
    Aug 19 22:21:52 server pure-ftpd: (user@x.x.x.x) [INFO] Logout.
    Aug 19 22:23:04 server pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
    Aug 19 22:23:04 server pure-ftpd: (?@127.0.0.1) [INFO] Logout.
    Aug 19 22:23:18 server pure-ftpd: (user@x.x.x.x) [NOTICE] /home/user//public_html/LatestNews.htm uploaded (56640 bytes, 6.15KB/sec)
    Aug 19 22:23:53 server kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000100
    Aug 19 22:23:53 server kernel: printing eip:
    Aug 19 22:23:53 server kernel: c0140b2b
    Aug 19 22:23:53 server kernel: *pde = 13ca9001
    Aug 19 22:23:53 server kernel: Oops: 0000 [#1]
    Aug 19 22:23:53 server kernel: SMP
    Aug 19 22:23:53 server kernel: Modules linked in: ipt_owner iptable_mangle ip_conntrack_ftp ipt_conntrack ipt_REJECT ipt_LOG ipt_limit ipt_multiport ipt_state ip_conntrack iptable_filter ip_tables md5 ipv6 autofs4 loop button battery ac uhci_hcd ehci_hcd e1000 dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod ata_piix libata sd_mod scsi_mod
    Aug 19 22:23:53 server kernel: CPU: 1
    Aug 19 22:23:53 server kernel: EIP: 0060:[<c0140b2b>] Not tainted VLI
    Aug 19 22:23:53 server kernel: EFLAGS: 00010097 (2.6.9-55.0.2.ELsmp)
    Aug 19 22:23:53 server kernel: EIP is at find_get_pages+0x30/0x50
    Aug 19 22:23:53 server kernel: eax: 4000112c ebx: 00000004 ecx: 00000003 edx: 00000100
    Aug 19 22:23:53 server kernel: esi: 0000000e edi: c87ae1a8 ebp: f6969f00 esp: f6969ec0
    Aug 19 22:23:53 server kernel: ds: 007b es: 007b ss: 0068
    Aug 19 22:23:53 server kernel: Process perl (pid: 25504, threadinfo=f6969000 task=f2256870)
    Aug 19 22:23:53 server kernel: Stack: 00000000 f6969ef8 ddcb2000 c87ae0f8 c01492e7 f6969f00 00000000 00000000
    Aug 19 22:23:53 server kernel: c01494fc 0000000e 00000246 00000000 00000000 c87ae1a8 00000000 00000000
    Aug 19 22:23:53 server kernel: c1c21dc0 c1c21de0 c1a0d800 00000100 f7e85e00 f8872ae1 cd920f1c f886b832
    Aug 19 22:23:53 server kernel: Call Trace:
    Aug 19 22:23:53 server kernel: [<c01492e7>] pagevec_lookup+0x17/0x1d
    Aug 19 22:23:53 server kernel: [<c01494fc>] truncate_inode_pages+0xbb/0x22d
    Aug 19 22:23:53 server kernel: [<f886b832>] __ext3_journal_stop+0x19/0x34 [ext3]
    Aug 19 22:23:53 server kernel: [<c0171fce>] generic_delete_inode+0x50/0x104
    Aug 19 22:23:53 server kernel: [<c0172203>] iput+0x5f/0x61
    Aug 19 22:23:53 server kernel: [<c0169304>] sys_unlink+0xd7/0x132
    Aug 19 22:23:53 server kernel: [<c02d6093>] syscall_call+0x7/0xb
    Aug 19 22:23:53 server kernel: [<c02d007b>] unix_stream_sendmsg+0x227/0x33a
    Aug 19 22:23:53 server kernel: Code: 40 10 56 89 d6 53 8b 6c 24 14 89 cb e8 a3 41 19 00 53 89 f1 8d 47 04 89 ea e8 8d 15 08 00 5e 89 c3 31 c9 39 d9 73 15 8b 54 8d 00 <8b> 02 f6 c4 80 74 03 8b 52 0c f0 ff 42 04 41 eb e7 8d 47 10 e8
    Aug 19 22:23:53 server kernel: <0>Fatal exception: panic in 5 seconds
    Aug 20 08:24:14 server syslogd 1.4.1: restart.
     
  3. cPanelNick

    cPanelNick Administrator
    Staff Member

    Joined:
    Mar 9, 2015
    Messages:
    3,426
    Likes Received:
    2
    Trophy Points:
    38
    cPanel Access Level:
    DataCenter Provider
    Take the server down into single user mode and fsck the file system.
     
  4. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    I did that but still happening...

    Sep 3 01:16:05 server kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000100
    Sep 3 01:16:05 server kernel: printing eip:
    Sep 3 01:16:05 server kernel: f88a016b
    Sep 3 01:16:05 server kernel: *pde = 34e36001
    Sep 3 01:16:05 server kernel: Oops: 0000 [#1]
    Sep 3 01:16:05 server kernel: SMP
    Sep 3 01:16:05 server kernel: Modules linked in: ipt_owner iptable_mangle ip_conntrack_ftp ipt_conntrack ipt_REJECT ipt_LOG ipt_limit ipt_multiport ipt_state ip_conntrack iptable_filter ip_tables md5 ipv6 autofs4 loop button battery ac uhci_hcd ehci_hcd e1000 dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod ata_piix libata sd_mod scsi_mod
    Sep 3 01:16:05 server kernel: CPU: 1
    Sep 3 01:16:05 server kernel: EIP: 0060:[<f88a016b>] Not tainted VLI
    Sep 3 01:16:05 server kernel: EFLAGS: 00010206 (2.6.9-55.0.2.ELsmp)
    Sep 3 01:16:05 server kernel: EIP is at __journal_remove_checkpoint+0xb/0x65 [jbd]
    Sep 3 01:16:05 server kernel: eax: f2571ecc ebx: 00000100 ecx: f2571ecc edx: f3365320
    Sep 3 01:16:05 server kernel: esi: f3365320 edi: c21f0e00 ebp: f3365320 esp: f7cfbe0c
    Sep 3 01:16:05 server kernel: ds: 007b es: 007b ss: 0068
    Sep 3 01:16:05 server kernel: Process kswapd0 (pid: 57, threadinfo=f7cfb000 task=f7d276f0)
    Sep 3 01:16:05 server kernel: Stack: f2571ecc f3365320 f889d973 f3365320 c11adc80 f2571ecc f889da19 00000000
    Sep 3 01:16:05 server kernel: c21f0e00 f88664d3 000000d0 c5e1e228 f7cfbf58 c015d5c7 c5e1e228 c11adc80
    Sep 3 01:16:05 server kernel: c0149e93 00000001 00000000 00000013 00000000 f7cfbec8 f7cfbe64 f7cfbe64
    Sep 3 01:16:05 server kernel: Call Trace:
    Sep 3 01:16:05 server kernel: [<f889d973>] __journal_try_to_free_buffer+0x69/0x89 [jbd]
    Sep 3 01:16:05 server kernel: [<f889da19>] journal_try_to_free_buffers+0x86/0xc5 [jbd]
    Sep 3 01:16:05 server kernel: [<f88664d3>] ext3_releasepage+0x0/0x54 [ext3]
    Sep 3 01:16:05 server kernel: [<c015d5c7>] try_to_release_page+0x34/0x46
    Sep 3 01:16:05 server kernel: [<c0149e93>] shrink_list+0x293/0x3ed
    Sep 3 01:16:05 server kernel: [<c014a1ca>] shrink_cache+0x1dd/0x34d
    Sep 3 01:16:05 server kernel: [<c014a888>] shrink_zone+0xa7/0xb6
    Sep 3 01:16:05 server kernel: [<c014ace7>] balance_pgdat+0x1c5/0x30e
    Sep 3 01:16:05 server kernel: [<c02d3f36>] schedule+0x87e/0x8ec
    Sep 3 01:16:05 server kernel: [<c0120458>] prepare_to_wait+0x12/0x4c
    Sep 3 01:16:05 server kernel: [<c014aefa>] kswapd+0xca/0xcc
    Sep 3 01:16:05 server kernel: [<c012052d>] autoremove_wake_function+0x0/0x2d
    Sep 3 01:16:05 server kernel: [<c02d5fae>] ret_from_fork+0x6/0x14
    Sep 3 01:16:05 server kernel: [<c012052d>] autoremove_wake_function+0x0/0x2d
    Sep 3 01:16:05 server kernel: [<c014ae30>] kswapd+0x0/0xcc
    Sep 3 01:16:05 server kernel: [<c01041f5>] kernel_thread_helper+0x5/0xb
    Sep 3 01:16:05 server kernel: Code: 0a 89 d8 e8 30 f9 ff ff 01 04 24 39 eb 75 e0 3b 74 24 08 75 c3 8b 04 24 83 c4 0c 5b 5e 5f 5d c3 56 89 c1 53 8b 58 24 85 db 74 57 <8b> 33 c7 40 24 00 00 00 00 8b 51 2c 8b 40 28 89 50 2c 8b 51 2c
    Sep 3 01:16:05 server kernel: <0>Fatal exception: panic in 5 seconds

    If its a file system problem I'm doomed - may have to move all accounts to a new server. Do you think its worth swapping the drive into an identical server to eliminate it being a hardware issue? I'm going to swap out the memory first to see if its just a faulty stick.
     
  5. astopy

    astopy Well-Known Member

    Joined:
    Apr 3, 2003
    Messages:
    165
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    Have you tried updating the kernel? It could just be a bug.
     
  6. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Unless it is a kernel bug, as Nick alluded to, this:
    suggests a file system problem, i.e. journaling failing on one of your ext3 partitions. If an FSCK doesn't fix it, you might have to replace the offending disk.
     
  7. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    Its a pain because the server can be fine for days then suddenly crash again. I've seen a very similar issue before but it was a bug on a previous kernel. All our servers are running the same kernel now and this is the only one with the problem so I've ruled that out.

    The fact that it seems to be the same issue every time would indicate that its not a memory problem. If it was memory, the debug info would be more random.

    I'm reluctant to think its a file system problem with it being a brand new drive, I'm using seagate barracuda ES drives in all my boxes and so far I've installed 52 of them and not had a single problem - but as you say, the debug info shows that at the time the server crashes, its trying to do some kind of swap file operation.

    Ok, the odds suggest that it is a file system problem because the server is performing the same file operation at the time it crashes - but I'm still not 100% sure and I dont want to go through the process of moving all these accounts to a new server, only to find the problem continues.

    I ran a FSCK and it didnt seem to find any issues, at least it didnt output that it found any problems or that it was fixing anything. I have kvmoip so I watched it running.

    One common factor is that there appears to be a lot of FTP activity at the time. I'm going to go back over the logs and see if there are any other trends I missed - i.e same user logged in, same files being uploaded etc.
     
Loading...

Share This Page