The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

APF crashing server constantly

Discussion in 'General Discussion' started by Secret Agent, Apr 7, 2005.

  1. Secret Agent

    Secret Agent Guest

    My server crashes about 5 times this week alone due to APF.

    Here is what I got back from my tech:

    After some examination, I have determined that the issues your server were experiencing were not hardware related. Initially, I booted the system into single user mode. The system ran nominally. Next, I started networking, and also examined your traffic graphs. As you said this was a recurring problem, I looked at your traffic over the past few weeks; nothing indicated a DoS or DDoS situation.

    Finally, upon bringing the system into multiuser mode, I noticed the server was taking an extraordinary amount of time to start 'apf', so much so that upon completing the boot cycle (several minutes later) the system was unusable. I rebooted the machine back into single user mode, disabled 'apf', and the iptables startup script and the system is now running as would be expected for the amount of users hosted on it.

    As I believed has been mentioned many times, you may want to reconsider using an automated firewalling tool. It seems to be causing you nothing but headaches. Also, bear in mind that while I disabled the init scripts for both apf and iptables, there is a chance that updating cPanel will cause them to be re-enabled. You may want to look into means for either paring down the ruleset that 'apf' is using, and configuring it to not firewall so aggresively. Unfortunately, as I am not intimately familiar with the use of this tool (only the havoc it can wreak), I cannot provide advice on how best to go about doing this.

    --

    I know it is APF as it has done this to my other servers.

    This is how it was installed / configured:

    wget http://www.rfxnetworks.com/downloads/apf-current.tar.gz
    tar -xvzf apf-current.tar.gz
    cd apf*
    ./install.sh
    pico /etc/apf/conf.apf

    FIND: DEVM="1"
    CHANGE TO: DEVM="0"

    Add to /etc/apf/conf.apf

    # Common ingress (inbound) TCP ports -3000_3500 = passive port range for Pure FTPD
    IG_TCP_CPORTS="21,22,25,53,80,110,143,443,2082,2083, 2086,2087, 2095, 2096,3000_3500"
    #
    # Common ingress (inbound) UDP ports
    IG_UDP_CPORTS="53"

    # Common egress (outbound) TCP ports
    EG_TCP_CPORTS="21,25,80,443,43,2089"
    #
    # Common egress (outbound) UDP ports
    EG_UDP_CPORTS="20,21,53"

    /usr/local/sbin/apf -s
    chkconfig --level 2345 apf on

    Specs:
    cPanel
    APF 0.9.5-1
    BFD
    LES
    Mod Security

    Please someone help me here. What should I do? I do not have mod_dosevasive nor ant-dos (from apf) enabled either.

    This is the original/untouched conf.apf file attached
     

    Attached Files:

    • apf.txt
      File size:
      11.5 KB
      Views:
      59
  2. gorilla

    gorilla Well-Known Member

    Joined:
    Feb 3, 2004
    Messages:
    699
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Sydney / Australia
    Never had any issues with APF on all our boxes , heard though that LES is problematic (not that i ever wanted to try it out ) ;)
     
  3. kris1351

    kris1351 Well-Known Member

    Joined:
    Apr 18, 2003
    Messages:
    963
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Lewisville, Tx
    Been using APF for over 2 years now on all of our servers and it has NEVER crashed a system. It would lock you out before it crashed you to be honest. The long time starting up sounds like it has a lot to filter through iptables. You could clear out the deny_hosts in APF and run iptables -F to make sure there is nothing corrupted there.
     
  4. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    The only problems I have seen with APF were actually to do with BFD v0.6 (which had some very serious bugs in it that could cause a server to lockout completely) - if you have that installed, I'd suggest that you upgrade it to BFD v0.7 asap and clear out your /etc/apf/deny_hosts.rules file completely. You should also check /etc/apf/ad/ad.rules to see how big that is, if you have antidos setup.
     
  5. Secret Agent

    Secret Agent Guest

    I upgraded to BFD 0.7 a while ago and clear my deny host rules often as well. Any other suggestions?

    How do I restart iptables again (tech says he disabled iptables startup script)

    I do not have antidos setup either.
     
  6. gorilla

    gorilla Well-Known Member

    Joined:
    Feb 3, 2004
    Messages:
    699
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Sydney / Australia
    I really thing you should be looking at your LES (Linux Environment Security) install
    Hey chirpy , didnt you have a fiddle with it a while ago ?
     
  7. Secret Agent

    Secret Agent Guest

    The odd thing is I have another server 100% identical in hardware /software / kernel / OS and yet that server never crashed before, has twice as much data on it and slightly higher average load.

    This is my typical server (both exactly same)

    Dual Xeons 2.4GHz HT
    2GB ECC Memory
    100Mbps Port
    APF | BFD | LES (all these 3 default settings)
    Mod Security
    Mod Throttle
    cPanel 10x1.x (Current)
    Fedora Core 2
    Kernel 2.6.9
     
  8. Secret Agent

    Secret Agent Guest

    LES is NOT installed anyway. I must of installed LES elsewhere to think it was here.

    So that rules out LES. Though it was installed, but isn't
     
    #8 Secret Agent, Apr 7, 2005
    Last edited by a moderator: Apr 7, 2005
  9. Aric1

    Aric1 Well-Known Member

    Joined:
    Oct 15, 2003
    Messages:
    324
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    DataCenter Provider
    Another thing you might want to look at is the ports you have open. Make sure you don't have ports closed that you need to have opened.

    Here's an example:

    Code:
    # Common ingress (inbound) TCP ports
    IG_TCP_CPORTS="20,21,22,25,26,53,80,110,143,443,465,873,993,995,2082,2083,2086,2087,2095,2096,3306,6666,9999,24441,2703,50000_50500"
    # Valid inbound TCP ports: 20 FTP, 21 FTP, 22 SSH, 25 SMTP, 26 SMTP-Alt. Port, 53 DNS, 80 HTTP, 110 POP3, 143 IMAP, 443 HTTPS, 465 SMTP-SSL, 873 RSYNC, 993 IMAP-SSL, 995 POP3-SSL, 2082 CPANEL, 2083 CPANEL-SSL, 2086 WHM, 2087 WHM-SSL, 2095 CPANEL Webmail, 2096 CPANEL Webmail-SSL, 3306 MySQL-Remote, 6666 Chat Service, 9999 Urchin Admin, 24441 Razor/Pyzor/DCC/SA, 2703 Razor/Pyzor/DCC/SA, 50000_50500 Passive FTP
    
    # Common ingress (inbound) UDP ports
    IG_UDP_CPORTS="21,53,465,873,24441"
    # Valid UDP inbound ports: 21 FTP, 53 DNS, 465 SMTP-SSL, 873 RSYNC, 24441 Razor/Pyzor/DCC/SA
    NOTE: Any spaces you see in the IG_TCP/UDP_CPORTS= lines are NOT really there. This forum is adding them and I can't get them removed.

    and outgoing:

    Code:
    # Common egress (outbound) TCP ports
    EG_TCP_CPORTS="20,21,22,25,26,37,43,53,80,113,443,465,873,2089,2703,3306,6666,24441,33434_33534"
    # Valid inbound TCP ports: 20 FTP, 21 FTP, 22 RSYNC, 25 SMTP, 26 SMTP-Alt.Port, 37 RDATE, 43 WHOIS, 53 DNS, 80 HTTP, 113 IDENT, 443 HTTPS, 465 SMTP-SSL, 873 RSYNC, 2089 CPANEL License Checking, 3306 MySQL Remote, 6666 Chat Service, 24441 Razor/Pyzor/DCC/SA, 2703 Razor/Pyzor/DCC/SA, 33434_33534 Traceroute
    
    # Common egress (outbound) UDP ports
    EG_UDP_CPORTS="21,53,465,873,6277,24441,33434_33534"
    # Valid UDP outbound ports: 21 FTP, 53 DNS, 465 SMTP-SSL, 873 RSYNC, 6277 Razor/Pyzor/DCC/SA, 24441 Razor/Pyzor/DCC/SA, 33434_33534 Traceroute
    NOTE: Any spaces you see in the EG_TCP/UDP_CPORTS= lines are NOT really there. This forum is adding them and I can't get them removed.

    This is just an example of what you might use. Obviously if you don't use Urchin or you don't have anything legitimate running on port 9999, you don't need to open that port, or perhaps you use different ports for passive FTP, etc..

    CPANEL and WHM do require a number of open ports to function correctly and not having them open could cause long delays for some services while they attempt to connect to a given port and fail.

    As I said, just another thing to consider.
     
    #9 Aric1, Apr 7, 2005
    Last edited: Apr 7, 2005
  10. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Didn't like it ;)

    Another thing to check with a crashing server is that you do not have laus installed:

    rpm -e laus

    This is another common cause of server crashes running cPanel.
     
  11. Secret Agent

    Secret Agent Guest

    error: package laus is not installed


    Strange. Any other possible suggestions? I checked TOP while the server load shot over 500 during (after) reboot, nothing in top abnormal, nor in cpu/memory/mysql usage as well
     
  12. rfxn

    rfxn Active Member

    Joined:
    Apr 27, 2003
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    APF , BFD or any other project from r-fx.org is NOT known to crash servers and has had and does have NO bugs that crash servers.

    With that said APF is a wrapper to create stateful packet inspection rules with iptables - iptables! is known to crash a server when kernel memory runs out, this can happen by having too many firewall rules and not enough memory - often this can be fixed by unloading many unneeded kernel modules that most people have inserted (soundcore, input, mousedev, keybrddev, ide-cdrom, usbcore, microcode, ppp, and so on).

    As well you can try change:
    # This is the maximum number of "sessions" (connection tracking entries)
    # that can be handled simultaneously by the firewall in kernel memory.
    # Increasing this value too high will simply waste memory; setting it
    # too low may result in some or all connections being refused, in paticular
    # during denial of service attacks.
    SYSCTL_CONNTRACK="34576"

    in /etc/apf/conf.apf - decrease value by about 10k and should help allocate bit more kernel memory to free space.
     
  13. Secret Agent

    Secret Agent Guest

    Is there a way to clear out the deny hosts rules entries via cron somehow?

    Every day I delete it two or three times, insane amount of entries
     
  14. gorilla

    gorilla Well-Known Member

    Joined:
    Feb 3, 2004
    Messages:
    699
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Sydney / Australia
    why do you delete them at all ? whenever an IP is attacking your servers you want this IP to be banned , dont yah ;)
     
  15. Secret Agent

    Secret Agent Guest

    Right but on a reboot it loads all these entires and locks up, even without reboots for some reason. I was told to delete them once in a while.
     
  16. gorilla

    gorilla Well-Known Member

    Joined:
    Feb 3, 2004
    Messages:
    699
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Sydney / Australia
    Are you sure your box is not comprimissed ?
     
  17. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    If you want to wipe your deny_hosts.rules (which as gorilla says is somewhat counter-productive) then you could simply run a cron job that copies /dev/null to the file at your own specified interval.
     
  18. kris1351

    kris1351 Well-Known Member

    Joined:
    Apr 18, 2003
    Messages:
    963
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Lewisville, Tx
    I would check on fixing the real problem instead of clearing APF every few hours. You might as well just delete it if you are just going to negate its actions. You have a real problem that is not the result of APF, you should hire someone to find it or get your DC to look into it.
     
  19. vwiley1

    vwiley1 Well-Known Member

    Joined:
    Oct 4, 2003
    Messages:
    87
    Likes Received:
    0
    Trophy Points:
    6
    Hi,

    I am having similar problems... I am hosting a domain that gets dictionary attacked very often.. A dictionary attack means a bunch of computers are trying to send email to EVERYNAME@example.com (aaa@example.com aaab@example.com ect....)

    Anyhow, the newest BFD can detect these types of attacks and add the IP's to APF. Well the problem is, is there is about 2000 new IP's per day, and even the IP's that are blocked....don't stop trying.

    When APF gets to many IP's built up... bandmin has trouble running (I get error emails telling me about memory allocation and also something about IPTABLES).. Anyhow, to correct the problem, I just reset the deny_hosts file, and everything goes back to normal.

    There is one problem I am still having though.... starting about midnight, and ending about 5 AM, my load shoots up to 5.00 to 6.00... My nightly updates and backup crons don't start till about 2 AM, so I know it's not them.

    The only thing I can tell from TOP, is that BFD starts, and before it ends another instance of BFD starts.. Anyhow it ends up with about 10 BFD's running and all taking up resources.
     
  20. kris1351

    kris1351 Well-Known Member

    Joined:
    Apr 18, 2003
    Messages:
    963
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Lewisville, Tx
    Try searching for the dictionary attack here on the forums. Chirpy did a wonderful write up on filtering it through Exim and it takes the load off of BFD and APF.
     
Loading...

Share This Page