kernel: Out of memory Killed Process xxxxx (httpd)

[rprix]

One of my servers is crashing everyday due high load caused by some httpd process that take all memory

This is a top out just a few minutes after restart apache:

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
5501 nobody 22 0 41900 40M 7256 R 11.0 2.0 0:29 2 httpd
5472 nobody 25 0 116M 115M 7412 R 9.1 5.7 0:39 1 httpd
5499 nobody 25 0 90108 87M 7388 R 9.1 4.3 0:33 1 httpd
5507 nobody 16 0 11416 10M 6960 R 5.3 0.5 0:19 0 httpd
5508 nobody 15 0 13140 12M 8112 S 5.3 0.6 0:22 3 httpd
5513 nobody 16 0 14772 13M 7668 R 3.0 0.6 0:15 1 httpd
6183 nobody 16 0 7912 7456 4384 R 2.8 0.3 0:00 0 httpd
6001 nobody 16 0 9516 9060 5588 R 2.5 0.4 0:07 2 httpd
5473 nobody 16 0 11200 10M 7184 R 2.3 0.5 0:17 0 httpd
5943 nobody 16 0 9928 9472 6024 R 2.1 0.4 0:05 1 httpd
...

normal httpd process are ~10mb, but some start taking all available memory (40mb, 115mb, 87mb ...) until server crash.

Some idea to found a solution to this and why some httpd process start taking all memory?

[brentp]

tail /usr/local/apache/logs/error_log

[rprix]

Nothing strange there, i just changed WarnLog to see more details or something strange

[rprix]

Nothing detected with WarnLog set at debug mode...
I disabled mod_gzip, mmturck and problem is still there.

Have other 5 or 6 servers with exactly same OS and software running (latest versions of everything) and this is only server with this problem.


#top
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
13555 nobody 25 0 171M 170M 7248 R 25.7 8.5 0:37 2 httpd

# ps -p 13555 -u
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
nobody 13555 13.6 10.4 237024 215316 ? R 17:05 0:47 /usr/local/apache/bin/httpd -DSSL

[dezignguy]

Have you thoroughly checked your system and made sure it's not one of the Perl worms that has been going around recently? (Most commonly through a nasty phpbb security hole, but certainly not limited to it). It was very common that they disguised themselves as Apache/httpd processes. And as they're scanning for and exploiting other vulnerable servers. they'll suck up copious amounts of cpu and ram. An apache child process shouldn't be getting that large, so that's kinda suspicous.

[cpanelnick]

Try /scripts/apachelimits