server hang

[cpanel_venkat]

Hi,
Server hangs,
Red Hat Linux 7.3,Kernel version is : 2.4.20-27.7
the log messages read as,


Jan 16 04:35:03 server2 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000070
Jan 16 04:35:03 server2 kernel: printing eip:
Jan 16 04:35:03 server2 kernel: c013b0c5
Jan 16 04:35:03 server2 kernel: *pde = 00000000
Jan 16 04:35:03 server2 kernel: Oops: 0000
Jan 16 04:35:03 server2 kernel: ipt_mark ipt_MARK ipt_TOS iptable_mangle ip_conntrack_ftp ip_conntrack_irc ipt_REJECT ipt_LOG ipt_limit ipt_unclean iptable_filter ipt_multiport ipt_state ip_
Jan 16 04:35:03 server2 kernel: CPU: 0
Jan 16 04:35:03 server2 kernel: EIP: 0010:[page_referenced+293/704] Not tainted
Jan 16 04:35:03 server2 kernel: EFLAGS: 00010216
Jan 16 04:35:03 server2 kernel:
Jan 16 04:35:03 server2 kernel: EIP is at page_referenced [kernel] 0x125 (2.4.20-27.7)
Jan 16 04:35:03 server2 kernel: eax: c3a43e80 ebx: 00000068 ecx: 00000000 edx: 005487ca
Jan 16 04:35:03 server2 kernel: esi: 00000163 edi: 00000005 ebp: c1d97810 esp: c1dfdeec
Jan 16 04:35:03 server2 kernel: ds: 0018 es: 0018 ss: 0018
Jan 16 04:35:03 server2 kernel: Process kswapd (pid: 5, stackpage=c1dfd000)
Jan 16 04:35:03 server2 kernel: Stack: 00000000 00000000 00000001 c1dfdf28 c1d9782c c1d97810 00000005 00000004
Jan 16 04:35:03 server2 kernel: c0132c12 000001f7 00000020 00000000 00000003 00000080 00005673 00000000
Jan 16 04:35:03 server2 kernel: 000005ee c02df650 0000015b 00000e17 c0134b24 c02df650 00000000 000000c0
Jan 16 04:35:03 server2 kernel: Call Trace: [refill_inactive_zone+802/4336] refill_inactive_zone [kernel] 0x322 (0xc1dfdf0c))
Jan 16 04:35:03 server2 kernel: [rebalance_inactive_zone+500/848] rebalance_inactive_zone [kernel] 0x1f4 (0xc1dfdf3c))
Jan 16 04:35:03 server2 kernel: [rebalance_inactive+61/128] rebalance_inactive [kernel] 0x3d (0xc1dfdf6c))
Jan 16 04:35:03 server2 kernel: [do_try_to_free_pages_kswapd+49/864] do_try_to_free_pages_kswapd [kernel] 0x31 (0xc1dfdf90))
Jan 16 04:35:03 server2 kernel: [kswapd+321/1248] kswapd [kernel] 0x141 (0xc1dfdfd4))
Jan 16 04:35:03 server2 kernel: [_stext+0/48] stext [kernel] 0x0 (0xc1dfdfe8))
Jan 16 04:35:03 server2 kernel: [arch_kernel_thread+38/48] arch_kernel_thread [kernel] 0x26 (0xc1dfdff0))
Jan 16 04:35:03 server2 kernel: [kswapd+0/1248] kswapd [kernel] 0x0 (0xc1dfdff8))
Jan 16 04:35:03 server2 kernel:
Jan 16 04:35:03 server2 kernel:
Jan 16 04:35:03 server2 kernel: Code: 8b 41 70 39 41 5c 0f 83 68 01 00 00 ff 44 24 04 e9 5f 01 00

I guess, it is memory related issue.
Any suggestions would be appreciated.

Thanks

[peruda.com]

I'm wondering if someone out there is found a vulnerability in cPanel/WHM recently. I seem to have been hacked as well.

At the bottom of this message is one of the e-mails I got just less than an hour ago. I also got three e-mails from the "[hackcheck]" script saying that the fileutils and net-tools RPMs were corrupted and the following files were modified:
S.5..UG. /bin/ls
S.5..UG. /usr/bin/dir
S.5..UG. /usr/bin/find
S.5..UG. /bin/netstat
S.5..UG. /sbin/ifconfig

Sure enough, the "ls" file is now owned by "proftpd" instead of root. Furthermore, when I attempt to force a reinstall of either the filetools or net-tools RPM, I get the error message, "error: unpacking of archive failed on file /bin/netstat: cpio: rename failed - Invalid argument" I also cannot chown any of the files.

I talked to a sys admin friend of mine and it sounds like a "rootkit" was run - possibly through proftp. In any case, your system may have been compromised and it is possible that the rootkit has installed a keystroke logger program that will e-mail your keystrokes including passwords etc. to the hacker. Our best bet might be to just reformat the server and restore from backup (of course, MAKE SURE you have a good backup on a second hard drive or another computer somewhere!)

-John


----- Original Message -----
Sent: Sunday, January 18, 2004 12:24 AM
Subject: [oopscheck] KERNEL Oops

IMPORTANT: Do not ignore this email.
Your kernel had an Oops!

This is the result of bad hardware or a kernel bug.
Your system may continue to function as normal, however
there is a good chance bad things are happening right now.
Bad things include: files disappearing, daemons crashing,
complete server crashs, disk corruption and many others.

You might want to check your RAM with memtest86 as this is
usually the cause of the problem.
http://www.memtest86.com/

The Oops is below:
Unable to handle kernel NULL pointer dereference at virtual address 0000003b
printing eip:
c00c6c00
*pde = 00000000
Oops: 0002
CPU: 0
EIP: 0010:[<c00c6c00>] Not tainted
EFLAGS: 00010297
eax: 0000003b ebx: c67f0000 ecx: 000000ff edx: 00000018
esi: c00c6c00 edi: 0804c863 ebp: bfffd31c esp: c67f1fc0
ds: 0018 es: 0018 ss: 0018
Process sk (pid: 26873, stackpage=c67f1000)
Stack: c01092cf 0000316d 000000ff 00000028 c00c6c00 0804c863 bfffd31c 0000003b
0000002b 0000002b 0000003b 080493a3 00000023 00000286 bfffd2b4 0000002b
Call Trace: [<c01092cf>]

Code: 00 00 00 83 ff 40 40 00 28 a3 ff a0 7f 10 ff 2b 05 ce 9f a0

[cpaneljosh]

Make sure you are running the latest kernel, there are security holes in redhat's kernel's below 2.4.20-28.X.

However: we have seen a lot of problem with 2.4.20-28.9, and recommend compiling 2.4.24 or 2.6.1 from source if possible.

[peruda.com]

It looks like I will be reformatting tonight and I figured I might as well switch from RH8 to FreeBSD 4.8 for increased security and because RH is discontinuing support. Does this make sense? Any recommendations on this?

Thanks!
-John

[DWHS.net]

Originally posted by cpaneljosh
Make sure you are running the latest kernel, there are security holes in redhat's kernel's below 2.4.20-28.X.

However: we have seen a lot of problem with 2.4.20-28.9, and recommend compiling 2.4.24 or 2.6.1 from source if possible.

You are not doing alerts on this anymore?

Which Kernel is the safest to upgrade to from 8.0?

Thanks..