The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Apache locks up due to hundreds of children

Discussion in 'Workarounds and Optimization' started by xnaspeed, Apr 19, 2012.

  1. xnaspeed

    xnaspeed Registered

    Joined:
    Apr 19, 2012
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    In the last day or so, my server has been having a hard time serving sites because apache has been locking up badly.

    Basically what happens is 200-300 apache processes start, and apache fails to serve.

    I don't know what's causing this exactly, although I have reason to believe it may be an attack on the server.

    ps aux | grep httpd show this (except 200-300 of these)

    Code:
    nobody   22498  0.0  0.0  64580  3920 ?        S    21:48   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   22499  0.0  0.0  64580  3896 ?        S    21:48   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   22500  0.0  0.0  64580  3956 ?        S    21:48   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   22501  0.0  0.0  64580  3892 ?        S    21:48   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   22502  0.0  0.0  64580  3892 ?        S    21:48   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   22579  0.0  0.0  64580  4028 ?        S    21:48   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   22621  0.0  0.0  64576  3680 ?        S    21:48   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   22632  0.0  0.0  64576  3680 ?        S    21:48   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   22633  0.0  0.0  64576  3796 ?        S    21:48   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   22634  0.0  0.0  64440  3116 ?        S    21:48   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    
    I have already made some changes, including KeepAlive settings I have seen in many other topics. I'm still waiting to see if that will actually help or not.

    Any other ideas are welcomed, as I'm still not sure what's actually causing it, and I'm not sure how to stop it from happening.

    Thanks.

    Edit: I should mention that during these "attacks", the processor and ram remain at average levels (usually below 0.5 load and 20% ram)
     
    #1 xnaspeed, Apr 19, 2012
    Last edited: Apr 19, 2012
  2. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
  3. xnaspeed

    xnaspeed Registered

    Joined:
    Apr 19, 2012
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    So running the netstat commands shows 557 apache processes, 256 in SYN_RECV, and 253 connections from one IP address.

    So I'm assuming this definitely means a DoS, correct?
     
  4. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Yes, it appears to be a DoS attack. You would want to block the one IP hitting the machine excessively after checking where that IP backtracks to:

    Code:
    dig -x IP#
    Please replace the IP number with the one in question. Hopefully, you copied down the IP number that was hitting the machine.
     
  5. NetMantis

    NetMantis BANNED

    Joined:
    Apr 22, 2012
    Messages:
    117
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Utah
    cPanel Access Level:
    DataCenter Provider
    Once you find out what IP(s) are attacking you, you can just simply block them.

    If you have CSF (highly recommended), you could block them in there either from the WHM interface or SSH commands.

    If you are just using plain old Linux without any special firewall, you could still block using IPTABLES:

    # iptables -A INPUT -s ##IP HERE## -j DROP

    You could also use route to dead null route the IP(s) or using the hosts file system though the later is a bit slower in response times but would still be useful.

    To do a hosts block, edit "/etc/hosts.deny" and put "ALL : ##IP HERE##" at the end of the file.

    Anyway, there is all sorts of different ways to block a bad IP from accessing your server.

    If you have mod_evasive or some of the other similar security modules out there, you could limit the number of simultaneous connections by any specific IP and have them self ban themselves temporarily opening up too many connections at once.

    Another item to also look into is your current Apache configuration as well as your server specs and make sure that your Apache configuration is properly tuned to handle enough connections and that you are not over running the physical capabilities of your server.

    Setup correctly, direct attacks on Apache would generally be fairly futile and pointless because Apache setup correctly can often handle the extra traffic no problem without slowing down. The issues most occur when Apache is not optimized or configured properly or the server is very weak low end hardware such as a VPS, etc in which case then yes you probably would be more vulnerable to direct web attacks.

    *EDIT* PS: I would avoid running PHP as DSO (mod_php). Aside from security issues, that really limits what you can see about what is going on at your server. If you are running SuPHP or FCGI, you will be much better equipped to see what all is being requested and running at any given moment on which account. Running "service httpd fullstatus" in the shell can also be quite useful as well.
     
    #5 NetMantis, Apr 23, 2012
    Last edited: Apr 23, 2012
Loading...

Share This Page