ARGH!!! Kernel Problems?!?!

[dominet]

Oct 17 02:22:38 host5 kernel: Out of Memory: Killed process 25328 (named).
Oct 17 02:22:38 host5 kernel: Out of Memory: Killed process 25330 (named).
Oct 17 02:22:38 host5 kernel: Out of Memory: Killed process 25331 (named).
Oct 17 02:22:38 host5 kernel: Out of Memory: Killed process 25332 (named).
Oct 17 02:22:38 host5 kernel: Out of Memory: Killed process 25333 (named).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 1371 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 1379 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 1380 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 976 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30344 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30620 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30660 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30722 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30825 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30828 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30905 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30979 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30980 (mysqld).
Oct 17 02:38:07 host5 kernel: Out of Memory: Killed process 30295 (httpd).


Oct 17 03:33:43 host5 kernel: Out of Memory: Killed process 30370 (httpd).


Oct 17 04:51:07 host5 kernel: ENOMEM in do_get_write_access, retrying.

 

[dominet]

2093 vipspowe 14 0 2344 2344 1136 R 0.9 0.4 0:00 spamassassin-cp
1215 root 11 0 1240 1240 828 R 0.8 0.2 0:00 top
2120 vipspowe 14 0 2200 2200 1136 R 0.7 0.4 0:00 spamassassin-cp
2122 vipspowe 14 0 2208 2208 1136 R 0.7 0.4 0:00 spamassassin-cp
2124 vipspowe 14 0 2200 2200 1136 R 0.7 0.4 0:00 spamassassin-cp
2126 vipspowe 14 0 2208 2208 1136 R 0.7 0.4 0:00 spamassassin-cp
2128 vipspowe 14 0 2208 2208 1136 R 0.7 0.4 0:00 spamassassin-cp
2131 vipspowe 14 0 2208 2208 1136 R 0.7 0.4 0:00 spamassassin-cp
2133 vipspowe 14 0 2208 2208 1136 R 0.7 0.4 0:00 spamassassin-cp
2135 vipspowe 14 0 2192 2192 1136 R 0.7 0.4 0:00 spamassassin-cp
2137 vipspowe 14 0 2208 2208 1136 R 0.7 0.4 0:00 spamassassin-cp
2139 vipspowe 14 0 2200 2200 1136 R 0.7 0.4 0:00 spamassassin-cp
2141 vipspowe 14 0 2148 2148 1136 R 0.7 0.4 0:00 spamassassin-cp
2143 vipspowe 14 0 2204 2204 1136 R 0.7 0.4 0:00 spamassassin-cp
2146 vipspowe 14 0 2212 2212 1136 R 0.7 0.4 0:00 spamassassin-cp
2148 vipspowe 14 0 2212 2212 1136 R 0.7 0.4 0:00 spamassassin-cp
2150 vipspowe 14 0 2212 2212 1136 R 0.7 0.4 0:00 spamassassin-cp
2152 vipspowe 14 0 2216 2216 1136 R 0.7 0.4 0:00 spamassassin-cp
2154 vipspowe 14 0 1996 1996 1132 R 0.7 0.3 0:00 spamassassin-cp
2187 vipspowe 20 0 2136 2136 1136 R 0.7 0.4 0:00 spamassassin-cp
2206 vipspowe 20 0 2268 2268 1136 R 0.7 0.4 0:00 spamassassin-cp
2208 vipspowe 20 0 2184 2184 1136 R 0.7 0.4 0:00 spamassassin-cp
2211 vipspowe 20 0 2212 2212 1136 R 0.7 0.4 0:00 spamassassin-cp
2213 vipspowe 20 0 2192 2192 1136 R 0.7 0.4 0:00 spamassassin-cp
2215 vipspowe 20 0 2212 2212 1136 R 0.7 0.4 0:00 spamassassin-cp
2217 vipspowe 20 0 2200 2200 1136 R 0.7 0.4 0:00 spamassassin-cp
2219 vipspowe 20 0 2200 2200 1136 R 0.7 0.4 0:00 spamassassin-cp
2221 vipspowe 20 0 2132 2132 1136 R 0.7 0.4 0:00 spamassassin-cp
2223 vipspowe 20 0 2124 2124 1136 R 0.7 0.4 0:00 spamassassin-cp
2226 vipspowe 20 0 2192 2192 1136 R 0.7 0.4 0:00 spamassassin-cp
2228 vipspowe 20 0 2388 2388 1136 R 0.7 0.4 0:00 spamassassin-cp
2230 vipspowe 20 0 2212 2212 1136 R 0.7 0.4 0:00 spamassassin-cp
2232 vipspowe 20 0 2156 2156 1136 R 0.7 0.4 0:00 spamassassin-cp
2236 vipspowe 20 0 2200 2200 1136 R 0.7 0.4 0:00 spamassassin-cp
2238 vipspowe 20 0 2040 2040 1136 R 0.7 0.3 0:00 spamassassin-cp
2240 vipspowe 20 0 2036 2036 1136 R 0.7 0.3 0:00 spamassassin-cp
2242 vipspowe 20 0 2100 2100 1136 R 0.7 0.4 0:00 spamassassin-cp
2244 vipspowe 20 0 2120 2120 1136 R 0.7 0.4 0:00 spamassassin-cp

seems thats the problem. Spamassassin again! What do do?

 

[silvernetuk]

Hi,

Originally posted by dominet
Oct 17 02:22:38 host5 kernel: Out of Memory: Killed process 25328 (named).
Oct 17 02:22:38 host5 kernel: Out of Memory: Killed process 25330 (named).
Oct 17 02:22:38 host5 kernel: Out of Memory: Killed process 25331 (named).
Oct 17 02:22:38 host5 kernel: Out of Memory: Killed process 25332 (named).
Oct 17 02:22:38 host5 kernel: Out of Memory: Killed process 25333 (named).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 1371 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 1379 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 1380 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 976 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30344 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30620 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30660 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30722 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30825 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30828 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30905 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30979 (mysqld).
Oct 17 02:32:51 host5 kernel: Out of Memory: Killed process 30980 (mysqld).
Oct 17 02:38:07 host5 kernel: Out of Memory: Killed process 30295 (httpd).


Oct 17 03:33:43 host5 kernel: Out of Memory: Killed process 30370 (httpd).


Oct 17 04:51:07 host5 kernel: ENOMEM in do_get_write_access, retrying.

Did you ever find out what this was caused by ?

Regards,
Garry

 

[MrHits]

I had a problem similar to this quite a long time ago with spam assasin. A user was getting flodded with mail, and his mail box was over enormous. I turned spamm assasin off for his account and it fixed it.

if it is the same issues, try running

ps -auxe

and see if this issue is tied to a single user, if it is, turn his spam assasin off, and make sure his mail box is not 2 gigs.

 

[parhelic]

This is very similar to what just happened on one of our servers tonight before it crashed:

Jan 26 20:06:55 cp10 kernel: <2>Page has mapping still set. This is a serious situation. However if you
Jan 26 20:06:55 cp10 kernel: are using the NVidia binary only module please report this bug to
Jan 26 20:06:55 cp10 kernel: NVidia and not to the linux kernel mailinglist.
Jan 26 20:06:55 cp10 kernel: ------------[ cut here ]------------
Jan 26 20:06:55 cp10 kernel: kernel BUG at page_alloc.c:122!
Jan 26 20:06:55 cp10 kernel: invalid operand: 0000
Jan 26 20:06:55 cp10 kernel: iptable_filter ip_tables parport_pc lp parport autofs e1000 loop keybdev mousedev hid input usb-uhci ehci-hcd usbcore ext3 jbd
Jan 26 20:06:55 cp10 kernel: CPU: 1
Jan 26 20:06:55 cp10 kernel: EIP: 0060:[] Not tainted
Jan 26 20:06:55 cp10 kernel: EFLAGS: 00010282
Jan 26 20:06:55 cp10 kernel:
Jan 26 20:06:55 cp10 kernel: EIP is at __free_pages_ok [kernel] 0x367 (2.4.20-28.9smp)
Jan 26 20:06:55 cp10 stunnel[1938]: Connection closed: 1727 bytes sent to SSL, 684 bytes sent to socket
Jan 26 20:06:55 cp10 kernel: eax: 00000033 ebx: c1c65870 ecx: 00000001 edx: f718c000
Jan 26 20:06:56 cp10 kernel: esi: ea8c0634 edi: 00000000 ebp: 00000000 esp: ceab3ed4
Jan 26 20:06:56 cp10 kernel: ds: 0068 es: 0068 ss: 0068
Jan 26 20:06:56 cp10 kernel: Process cpanel (pid: 5277, stackpage=ceab3000)
Jan 26 20:06:56 cp10 kernel: Stack: c0286e00 42d78025 080ff000 ffffffff 000000ff 080ff000 c01397b0 cd2f791c
Jan 26 20:06:56 cp10 kernel: c4fdb180 c259dd80 c1c65870 00000100 08248000 00000100 c0136bdc c1c65870
Jan 26 20:06:56 cp10 kernel: 000000b6 c0137455 c4fdb180 d534a080 08000000 00100000 08548000 ceab2000
Jan 26 20:06:56 cp10 kernel: Call Trace: [] zap_pte_range [kernel] 0x160 (0xceab3eec))
Jan 26 20:06:56 cp10 kernel: [] __free_pte [kernel] 0x4c (0xceab3f0c))
Jan 26 20:06:56 cp10 kernel: [] zap_page_range [kernel] 0x1a5 (0xceab3f18))
Jan 26 20:06:56 cp10 kernel: [] exit_mmap [kernel] 0xd0 (0xceab3f5c))
Jan 26 20:06:56 cp10 kernel: [] mmput [kernel] 0x62 (0xceab3f80))
Jan 26 20:06:56 cp10 kernel: [] do_exit [kernel] 0x136 (0xceab3f90))
Jan 26 20:06:56 cp10 kernel: [] do_group_exit [kernel] 0x8b (0xceab3fac))
Jan 26 20:06:56 cp10 kernel: [] system_call [kernel] 0x33 (0xceab3fc0))
Jan 26 20:06:56 cp10 kernel:
Jan 26 20:06:56 cp10 kernel:
Jan 26 20:06:56 cp10 kernel: Code: 0f 0b 7a 00 da 65 28 c0 e9 e5 fc ff ff 0f 0b 69 00 da 65 28


Then further down in message I see this:

Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 5306 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 5924 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 5951 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 5969 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 5971 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 5972 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 5983 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 6013 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 6017 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 6019 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 6022 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 2368 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 2369 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 15188 (mysqld).
Jan 26 20:07:31 cp10 kernel: Out of Memory: Killed process 10044 (mysqld).
Jan 26 20:26:07 cp10 kernel: Out of Memory: Killed process 11634 (mysqld).
Jan 26 20:26:07 cp10 kernel: Out of Memory: Killed process 11635 (mysqld).
Jan 26 20:26:12 cp10 kernel: Out of Memory: Killed process 8213 (httpd).
Jan 26 20:26:17 cp10 kernel: Out of Memory: Killed process 8225 (httpd).
Jan 26 20:26:22 cp10 kernel: Out of Memory: Killed process 8227 (httpd).
Jan 26 20:26:27 cp10 kernel: Out of Memory: Killed process 10613 (httpd).
Jan 26 20:26:32 cp10 kernel: Out of Memory: Killed process 8221 (httpd).
Jan 26 20:26:37 cp10 kernel: Out of Memory: Killed process 8217 (httpd).
Jan 26 20:26:42 cp10 kernel: Out of Memory: Killed process 8215 (httpd).
Jan 26 20:26:47 cp10 kernel: Out of Memory: Killed process 8219 (httpd).


I really have no idea how to troubleshoot the problem, could it be hardware, or something else?