APF crashing server constantly

[Secret Agent]

My server crashes about 5 times this week alone due to APF.

Here is what I got back from my tech:

After some examination, I have determined that the issues your server were experiencing were not hardware related. Initially, I booted the system into single user mode. The system ran nominally. Next, I started networking, and also examined your traffic graphs. As you said this was a recurring problem, I looked at your traffic over the past few weeks; nothing indicated a DoS or DDoS situation.

Finally, upon bringing the system into multiuser mode, I noticed the server was taking an extraordinary amount of time to start 'apf', so much so that upon completing the boot cycle (several minutes later) the system was unusable. I rebooted the machine back into single user mode, disabled 'apf', and the iptables startup script and the system is now running as would be expected for the amount of users hosted on it.

As I believed has been mentioned many times, you may want to reconsider using an automated firewalling tool. It seems to be causing you nothing but headaches. Also, bear in mind that while I disabled the init scripts for both apf and iptables, there is a chance that updating cPanel will cause them to be re-enabled. You may want to look into means for either paring down the ruleset that 'apf' is using, and configuring it to not firewall so aggresively. Unfortunately, as I am not intimately familiar with the use of this tool (only the havoc it can wreak), I cannot provide advice on how best to go about doing this.

--

I know it is APF as it has done this to my other servers.

This is how it was installed / configured:

wget http://www.rfxnetworks.com/downloads/apf-current.tar.gz
tar -xvzf apf-current.tar.gz
cd apf*
./install.sh
pico /etc/apf/conf.apf

FIND: DEVM="1"
CHANGE TO: DEVM="0"

Add to /etc/apf/conf.apf

# Common ingress (inbound) TCP ports -3000_3500 = passive port range for Pure FTPD
IG_TCP_CPORTS="21,22,25,53,80,110,143,443,2082,2083, 2086,2087, 2095, 2096,3000_3500"
#
# Common ingress (inbound) UDP ports
IG_UDP_CPORTS="53"

# Common egress (outbound) TCP ports
EG_TCP_CPORTS="21,25,80,443,43,2089"
#
# Common egress (outbound) UDP ports
EG_UDP_CPORTS="20,21,53"

/usr/local/sbin/apf -s
chkconfig --level 2345 apf on

Specs:
cPanel
APF 0.9.5-1
BFD
LES
Mod Security

Please someone help me here. What should I do? I do not have mod_dosevasive nor ant-dos (from apf) enabled either.

This is the original/untouched conf.apf file attached

 

[gorilla]

Never had any issues with APF on all our boxes , heard though that LES is problematic (not that i ever wanted to try it out )

 

[kris1351]

Been using APF for over 2 years now on all of our servers and it has NEVER crashed a system. It would lock you out before it crashed you to be honest. The long time starting up sounds like it has a lot to filter through iptables. You could clear out the deny_hosts in APF and run iptables -F to make sure there is nothing corrupted there.

 

[chirpy]

The only problems I have seen with APF were actually to do with BFD v0.6 (which had some very serious bugs in it that could cause a server to lockout completely) - if you have that installed, I'd suggest that you upgrade it to BFD v0.7 asap and clear out your /etc/apf/deny_hosts.rules file completely. You should also check /etc/apf/ad/ad.rules to see how big that is, if you have antidos setup.

 

[Secret Agent]

I upgraded to BFD 0.7 a while ago and clear my deny host rules often as well. Any other suggestions?

How do I restart iptables again (tech says he disabled iptables startup script)

I do not have antidos setup either.

 

[gorilla]

I really thing you should be looking at your LES (Linux Environment Security) install
Hey chirpy , didnt you have a fiddle with it a while ago ?

 

[Secret Agent]

The odd thing is I have another server 100% identical in hardware /software / kernel / OS and yet that server never crashed before, has twice as much data on it and slightly higher average load.

This is my typical server (both exactly same)

Dual Xeons 2.4GHz HT
2GB ECC Memory
100Mbps Port
APF | BFD | LES (all these 3 default settings)
Mod Security
Mod Throttle
cPanel 10x1.x (Current)
Fedora Core 2
Kernel 2.6.9

 

[Secret Agent]

LES is NOT installed anyway. I must of installed LES elsewhere to think it was here.

So that rules out LES. Though it was installed, but isn't

 

[Aric1]

Another thing you might want to look at is the ports you have open. Make sure you don't have ports closed that you need to have opened.

Here's an example:

# Common ingress (inbound) TCP ports
IG_TCP_CPORTS="20,21,22,25,26,53,80,110,143,443,465,873,993,995,2082,2083,2086,2087,2095,2096,3306,6666,9999,24441,2703,50000_50500"
# Valid inbound TCP ports: 20 FTP, 21 FTP, 22 SSH, 25 SMTP, 26 SMTP-Alt. Port, 53 DNS, 80 HTTP, 110 POP3, 143 IMAP, 443 HTTPS, 465 SMTP-SSL, 873 RSYNC, 993 IMAP-SSL, 995 POP3-SSL, 2082 CPANEL, 2083 CPANEL-SSL, 2086 WHM, 2087 WHM-SSL, 2095 CPANEL Webmail, 2096 CPANEL Webmail-SSL, 3306 MySQL-Remote, 6666 Chat Service, 9999 Urchin Admin, 24441 Razor/Pyzor/DCC/SA, 2703 Razor/Pyzor/DCC/SA, 50000_50500 Passive FTP

# Common ingress (inbound) UDP ports
IG_UDP_CPORTS="21,53,465,873,24441"
# Valid UDP inbound ports: 21 FTP, 53 DNS, 465 SMTP-SSL, 873 RSYNC, 24441 Razor/Pyzor/DCC/SA

NOTE: Any spaces you see in the IG_TCP/UDP_CPORTS= lines are NOT really there. This forum is adding them and I can't get them removed.

and outgoing:

# Common egress (outbound) TCP ports
EG_TCP_CPORTS="20,21,22,25,26,37,43,53,80,113,443,465,873,2089,2703,3306,6666,24441,33434_33534"
# Valid inbound TCP ports: 20 FTP, 21 FTP, 22 RSYNC, 25 SMTP, 26 SMTP-Alt.Port, 37 RDATE, 43 WHOIS, 53 DNS, 80 HTTP, 113 IDENT, 443 HTTPS, 465 SMTP-SSL, 873 RSYNC, 2089 CPANEL License Checking, 3306 MySQL Remote, 6666 Chat Service, 24441 Razor/Pyzor/DCC/SA, 2703 Razor/Pyzor/DCC/SA, 33434_33534 Traceroute

# Common egress (outbound) UDP ports
EG_UDP_CPORTS="21,53,465,873,6277,24441,33434_33534"
# Valid UDP outbound ports: 21 FTP, 53 DNS, 465 SMTP-SSL, 873 RSYNC, 6277 Razor/Pyzor/DCC/SA, 24441 Razor/Pyzor/DCC/SA, 33434_33534 Traceroute

NOTE: Any spaces you see in the EG_TCP/UDP_CPORTS= lines are NOT really there. This forum is adding them and I can't get them removed.

This is just an example of what you might use. Obviously if you don't use Urchin or you don't have anything legitimate running on port 9999, you don't need to open that port, or perhaps you use different ports for passive FTP, etc..

CPANEL and WHM do require a number of open ports to function correctly and not having them open could cause long delays for some services while they attempt to connect to a given port and fail.

As I said, just another thing to consider.

 

[chirpy]

Hey chirpy , didnt you have a fiddle with it a while ago ?

Didn't like it

Another thing to check with a crashing server is that you do not have laus installed:

rpm -e laus

This is another common cause of server crashes running cPanel.

 

[Secret Agent]

error: package laus is not installed


Strange. Any other possible suggestions? I checked TOP while the server load shot over 500 during (after) reboot, nothing in top abnormal, nor in cpu/memory/mysql usage as well

 

[rfxn]

APF , BFD or any other project from r-fx.org is NOT known to crash servers and has had and does have NO bugs that crash servers.

With that said APF is a wrapper to create stateful packet inspection rules with iptables - iptables! is known to crash a server when kernel memory runs out, this can happen by having too many firewall rules and not enough memory - often this can be fixed by unloading many unneeded kernel modules that most people have inserted (soundcore, input, mousedev, keybrddev, ide-cdrom, usbcore, microcode, ppp, and so on).

As well you can try change:
# This is the maximum number of "sessions" (connection tracking entries)
# that can be handled simultaneously by the firewall in kernel memory.
# Increasing this value too high will simply waste memory; setting it
# too low may result in some or all connections being refused, in paticular
# during denial of service attacks.
SYSCTL_CONNTRACK="34576"

in /etc/apf/conf.apf - decrease value by about 10k and should help allocate bit more kernel memory to free space.

 

[Secret Agent]

Is there a way to clear out the deny hosts rules entries via cron somehow?

Every day I delete it two or three times, insane amount of entries

 

[gorilla]

why do you delete them at all ? whenever an IP is attacking your servers you want this IP to be banned , dont yah

 

[Secret Agent]

Right but on a reboot it loads all these entires and locks up, even without reboots for some reason. I was told to delete them once in a while.

 

[gorilla]

Are you sure your box is not comprimissed ?

 

[chirpy]

If you want to wipe your deny_hosts.rules (which as gorilla says is somewhat counter-productive) then you could simply run a cron job that copies /dev/null to the file at your own specified interval.

 

[kris1351]

I would check on fixing the real problem instead of clearing APF every few hours. You might as well just delete it if you are just going to negate its actions. You have a real problem that is not the result of APF, you should hire someone to find it or get your DC to look into it.

 

[vwiley1]

Hi,

I am having similar problems... I am hosting a domain that gets dictionary attacked very often.. A dictionary attack means a bunch of computers are trying to send email to [email protected] ([email protected] [email protected] ect....)

Anyhow, the newest BFD can detect these types of attacks and add the IP's to APF. Well the problem is, is there is about 2000 new IP's per day, and even the IP's that are blocked....don't stop trying.

When APF gets to many IP's built up... bandmin has trouble running (I get error emails telling me about memory allocation and also something about IPTABLES).. Anyhow, to correct the problem, I just reset the deny_hosts file, and everything goes back to normal.

There is one problem I am still having though.... starting about midnight, and ending about 5 AM, my load shoots up to 5.00 to 6.00... My nightly updates and backup crons don't start till about 2 AM, so I know it's not them.

The only thing I can tell from TOP, is that BFD starts, and before it ends another instance of BFD starts.. Anyhow it ends up with about 10 BFD's running and all taking up resources.

 

[kris1351]

Try searching for the dictionary attack here on the forums. Chirpy did a wonderful write up on filtering it through Exim and it takes the load off of BFD and APF.