apache failing every 3 hours

[totalufo]

This is the second server that started doing this. put up 2 new servers and both fail apache about every 3 hours. Any ideas?

 

[feanor]

SYN flood perhaps against your entire network perhaps?
Or you need to increase max connections on your webservers....

Check /var/log/messages and netstat -an for hints on whom is assaulting you on port 80- that could definitely be a possibility. Otherwise your connections are maxing out and apache is hanging once it runs out of memory for everybody that you allowed it to listen for

 

[totalufo]

This guy did actually put about 400 domains on this server


Here is the netstat -an
sorry for the huge list

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:2593 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:1 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2082 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2083 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:995 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2084 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2086 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2087 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:7786 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6666 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2092 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:110 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2095 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2096 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:465 0.0.0.0:* LISTEN
tcp 0 0 209.51.131.174:53 0.0.0.0:* LISTEN
tcp 0 0 209.51.131.173:53 0.0.0.0:* LISTEN
tcp 0 0 209.51.131.172:53 0.0.0.0:* LISTEN
tcp 0 0 209.51.131.171:53 0.0.0.0:* LISTEN
tcp 0 0 209.51.131.170:53 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN
tcp 0 0 209.51.131.170:80 198.81.8.2:62590 TIME_WAIT
tcp 0 8 209.51.131.170:2593 200.176.88.69:1346 ESTABLISHED
tcp 0 0 209.51.131.170:80 198.81.8.3:52862 TIME_WAIT
tcp 0 16 209.51.131.170:2593 200.210.249.102:1056 ESTABLISHED
tcp 0 0 209.51.131.170:80 198.81.8.148:57364 TIME_WAIT
tcp 0 0 209.51.131.170:2086 200.168.40.148:1026 ESTABLISHED
tcp 0 6246 209.51.131.170:80 200.170.214.34:53246 CLOSING
tcp 0 0 209.51.131.170:2082 200.168.40.148:1025 FIN_WAIT2
tcp 0 0 209.51.131.170:2086 200.168.40.148:1025 TIME_WAIT
tcp 0 0 209.51.131.170:21 172.187.33.26:1773 ESTABLISHED
tcp 0 0 209.51.131.170:25 200.188.191.51:4575 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.168.40.148:1032 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.168.40.148:1033 TIME_WAIT
tcp 0 0 209.51.131.170:80 200.168.40.148:1025 ESTABLISHED
tcp 0 0 209.51.131.170:80 198.81.8.2:62897 TIME_WAIT
tcp 0 0 209.51.131.170:2593 200.195.174.15:1372 ESTABLISHED
tcp 0 0 209.51.131.170:80 198.81.8.1:21548 TIME_WAIT
tcp 0 2882 209.51.131.170:80 200.151.0.235:1716 ESTABLISHED
tcp 0 0 209.51.131.170:2593 200.216.10.97:1359 ESTABLISHED
tcp 0 121 209.51.131.170:2082 200.195.123.42:2603 ESTABLISHED
tcp 0 0 209.51.131.170:80 200.151.0.235:1715 ESTABLISHED
tcp 0 4593 209.51.131.170:2082 200.195.123.42:2602 ESTABLISHED
tcp 0 325 209.51.131.170:2082 200.195.123.42:2601 ESTABLISHED
tcp 0 0 209.51.131.170:80 200.195.123.42:2593 TIME_WAIT
tcp 0 148 209.51.131.170:2082 200.195.123.42:2600 ESTABLISHED
tcp 0 0 209.51.131.170:80 198.81.8.1:21674 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.195.123.42:2599 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.195.123.42:2598 TIME_WAIT
tcp 0 56 209.51.131.170:25 200.168.71.82:1025 ESTABLISHED
tcp 0 0 209.51.131.170:2082 200.195.123.42:2597 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.216.10.97:1540 FIN_WAIT2
tcp 0 0 209.51.131.170:2082 200.195.123.42:2596 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.195.123.42:2595 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.195.123.42:2594 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.216.10.97:1539 FIN_WAIT2
tcp 0 0 209.51.131.170:80 198.81.8.2:62817 TIME_WAIT
tcp 0 0 209.51.131.170:110 200.242.76.65:1029 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.216.10.97:1537 FIN_WAIT2
tcp 0 0 209.51.131.170:25 200.170.63.188:1043 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.226.91.61:1481 TIME_WAIT
tcp 0 11 209.51.131.170:2593 200.176.85.125:1047 ESTABLISHED
tcp 0 0 209.51.131.170:2082 200.226.91.61:1483 TIME_WAIT
tcp 0 0 209.51.131.170:80 198.81.8.1:21592 TIME_WAIT
tcp 0 0 209.51.131.170:110 200.221.36.197:2243 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.226.91.61:1487 TIME_WAIT
tcp 0 0 209.51.131.170:80 198.81.9.3:28757 TIME_WAIT
tcp 0 143 209.51.131.170:2593 172.187.88.45:1034 ESTABLISHED
tcp 0 2 209.51.131.170:2593 200.193.165.42:1037 ESTABLISHED
tcp 0 0 209.51.131.170:80 198.81.9.3:28499 TIME_WAIT
tcp 0 2 209.51.131.170:2593 200.193.165.42:1038 ESTABLISHED
tcp 0 0 209.51.131.170:110 200.220.32.96:2568 TIME_WAIT
tcp 0 0 209.51.131.170:80 198.81.9.2:53901 TIME_WAIT
tcp 0 4512 209.51.131.170:22 4.64.25.151:4903 ESTABLISHED
tcp 0 59 209.51.131.170:2593 200.191.188.171:1101 ESTABLISHED
tcp 0 0 209.51.131.170:2082 200.226.91.61:1489 ESTABLISHED
tcp 0 0 209.51.131.170:2082 200.226.91.61:1488 TIME_WAIT
tcp 0 0 209.51.131.170:2082 200.226.91.61:1491 TIME_WAIT
tcp 0 0 209.51.131.170:2095 200.178.101.145:6318 ESTABLISHED
tcp 0 0 209.51.131.170:2082 200.226.91.61:1490 ESTABLISHED
tcp 0 0 209.51.131.170:25 200.231.206.186:37106 ESTABLISHED
tcp 0 0 209.51.131.170:80 198.81.8.2:62658 TIME_WAIT
tcp 0 0 209.51.131.170:80 198.81.9.3:28803 TIME_WAIT
tcp 0 0 209.51.131.170:110 200.220.32.96:2564 TIME_WAIT
udp 0 0 0.0.0.0:34339 0.0.0.0:*
udp 0 0 209.51.131.174:53 0.0.0.0:*
udp 0 0 209.51.131.173:53 0.0.0.0:*
udp 0 0 209.51.131.172:53 0.0.0.0:*
udp 0 0 209.51.131.171:53 0.0.0.0:*
udp 0 0 209.51.131.170:53 0.0.0.0:*
udp 0 0 127.0.0.1:53 0.0.0.0:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 72713 /usr/local/cpanel/3rdparty/interchange/etc/socket
unix 13 [ ] DGRAM 1195 /dev/log
unix 2 [ ACC ] STREAM LISTENING 1676769 /var/lib/mysql/mysql.sock
unix 2 [ ACC ] STREAM LISTENING 72711 /usr/local/cpanel/3rdparty/interchange/etc/socket.ipc
unix 2 [ ] DGRAM 1682232
unix 2 [ ] DGRAM 1665283
unix 2 [ ] DGRAM 72684
unix 2 [ ] DGRAM 72682
unix 2 [ ] DGRAM 72677
unix 2 [ ] DGRAM 72658
unix 2 [ ] DGRAM 72655
unix 2 [ ] DGRAM 1947
unix 2 [ ] DGRAM 1420
unix 2 [ ] DGRAM 1364
unix 2 [ ] DGRAM 1204
unix 2 [ ] STREAM CONNECTED 633

 

[feanor]

Not helpful really.
You need to do some more of that combined with tailing apache logs, and analyzing bandwidth to determine who is actually doing damage, and against what specifically.

Or you definitely need to increase MaxClients within your httpd.conf(s), if that is part or all of the issue.

Or....... ?
Just need to dig up some more details, it appears.

 

[totalufo]

ok, I stuck a monitor on this server and tailed the /var/log/messages when apache failed

Here's what I got. I think you might be right

Mar 12 03:10:45 localhost stunnel[8543]: stunnel 3.22 on i686-pc-linux-gnu PTHREAD+LIBWRAP with OpenSSL 0.9.5a 1 Apr 2000
Mar 12 03:10:45 localhost stunnel[8545]: Using '127.0.0.1.2095' as tcpwrapper service name
Mar 12 03:10:45 localhost stunnel[8544]: FD_SETSIZE=16384, file ulimit=1024 -& 500 clients allowed
Mar 12 03:10:45 localhost stunnel[8542]: FD_SETSIZE=16384, file ulimit=1024 -& 500 clients allowed
Mar 12 03:10:45 localhost stunnel[8545]: stunnel 3.22 on i686-pc-linux-gnu PTHREAD+LIBWRAP with OpenSSL 0.9.5a 1 Apr 2000
Mar 12 03:10:45 localhost stunnel[8546]: FD_SETSIZE=16384, file ulimit=1024 -& 500 clients allowed

 

[feanor]

/etc/rc.d/init.d/filelimits start &
Do that a couple times actually....

And then increase your MaxClients and StartServers in your httpd.conf to something reasonable.

Should be golden then.

 

[totalufo]

Here's what I get.

Increasing file system limits [ OK ]
/etc/rc.d/init.d/filelimits: /proc/sys/fs/dquot-max: No such file or directory

 

[feanor]

Ah you must have RH 7.2
Since we've had the ability for that OS version on cpanel machines..... filelimits seem to be controlled elsewhere.....

I am hoping someone else on the forum may have some input on this, as this is something I have always taken for granted- having that init script I always relied upon it until now.

Anyone else have some insight on the most efficient ways to increase file descriptors within RH 7.2 ?


Thanks...

 

[totalufo]

After increasing the maxx connections to the highest/safest levels, it still fails. It has to be something else as well

 

[bdraco]

[quote:03ff85730e][i:03ff85730e]Originally posted by totalufo[/i:03ff85730e]

After increasing the maxx connections to the highest/safest levels, it still fails. It has to be something else as well[/quote:03ff85730e]

Check the /usr/local/apache/logs/error_log

 

[totalufo]

Here's what I get during a fail

[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_env.so uses plain Apache 1.3 API, this module might c$
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_log_config.so uses plain Apache 1.3 API, this module $
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_mime.so uses plain Apache 1.3 API, this module might $
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_negotiation.so uses plain Apache 1.3 API, this module$
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_status.so uses plain Apache 1.3 API, this module migh$
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_include.so uses plain Apache 1.3 API, this module mig$
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_autoindex.so uses plain Apache 1.3 API, this module m$
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_dir.so uses plain Apache 1.3 API, this module might c$
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_cgi.so uses plain Apache 1.3 API, this module might c$
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_asis.so uses plain Apache 1.3 API, this module might $
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_imap.so uses plain Apache 1.3 API, this module might $
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_actions.so uses plain Apache 1.3 API, this module mig$
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_userdir.so uses plain Apache 1.3 API, this module mig$
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_alias.so uses plain Apache 1.3 API, this module might$
[Sat Mar 2 12:55:36 2002] [warn] Loaded DSO libexec/mod_access.so uses plain Apache 1.3 API, this module migh$

 

[totalufo]

Ah HAH! Found the problem so it seams. Now just a matter of fixing it.

[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_userdir.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)

Here's the entire tail of the error log

[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_autoindex.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_dir.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_cgi.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_asis.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_imap.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_actions.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_userdir.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_alias.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
httpd: module &mod_alias.c& could not be loaded, because the dynamic
module limit was reached. Please increase DYNAMIC_MODULE_LIMIT and recompile.

 

[bdraco]

[quote:0789a1ea2f][i:0789a1ea2f]Originally posted by totalufo[/i:0789a1ea2f]

Ah HAH! Found the problem so it seams. Now just a matter of fixing it.

[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_userdir.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)

Here's the entire tail of the error log

[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_autoindex.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_dir.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_cgi.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_asis.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_imap.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_actions.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_userdir.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
[Wed Mar 13 01:00:05 2002] [warn] Loaded DSO libexec/mod_alias.so uses plain Apache 1.3 API, this module might crash under EAPI! (please recompile it with -DEAPI)
httpd: module &mod_alias.c& could not be loaded, because the dynamic
module limit was reached. Please increase DYNAMIC_MODULE_LIMIT and recompile.[/quote:0789a1ea2f]

copy all the vhosts to a diffrent file.. move the httpd.conf to somewhere else reinstall buildapache.sea. Put the vhosts on the bottom of the new httpd.conf it creates. restart httpd.