The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

strange HTTPD DoS attack ...

Discussion in 'General Discussion' started by student, Sep 26, 2006.

  1. student

    student Active Member

    Joined:
    Aug 9, 2006
    Messages:
    38
    Likes Received:
    0
    Trophy Points:
    6
    Sometimes, my apache web server is getting overloaded by unknown requests. During one of this strange overload, using WHM -> 'Server Status'->'Apache Status' I've noticed enormous amount of unusual connections:
    eg.
    51-7 | 3408 | 0/97/2961 | R | 0.47 | 43 | 1 |0.0 | 0.93 | 80.44 | ? | ? | ..reading..
    ...
    so we have in the request:
    Host: ? Vhost: ? Request: ..reading..

    and because of that connections, normal sites are unavailable (very, very long loading time, or timeout)

    any suggestions how to deal with this?
    btw. after apache restart (service httpd restart) all is ok again, I see usual connecions by normal users, without strange '..reading..' requests...

    Is there any way to see the output of WHM -> 'Server Status'->'Apache Status' in the shell? I mean is there some kind of shell script that does that and producing output not to WHM, but my shell console?

    Ok, and the last question, I would like to ask about SWAP space. Seeing lots of sudden 'load average' overload jumpings, and [kswapd0] process eating all of my cpu%. I've decided to turn off SWAP space. Anyone has experience with non SWAP space server working? Is it good solution? I've 2GB of RAM, and in my opinion, SWAP is not needed in such ampunt of RAM (that is max. used in ~50%)

    Appreciate any help, thanks
     
  2. Manuel_accu

    Manuel_accu Well-Known Member

    Joined:
    Jun 19, 2005
    Messages:
    191
    Likes Received:
    0
    Trophy Points:
    16
  3. driverC

    driverC Well-Known Member

    Joined:
    Jul 23, 2004
    Messages:
    112
    Likes Received:
    0
    Trophy Points:
    16
    I had the same type of attack quite often. They connect to Apache to create a process and then they don't do anything leaving the process open for as long as it takes to time out. Often they connect to the server's IP so it is impossible to stop the attack by suspending a specific account. As a result of this Apache may reach the process limit and reject new connections or the server may break down overloaded just because of some 14 year old little ****.

    Here is how to deal with it:
    1) SSH into your server
    2) Enter this command: pico /usr/local/apache/conf/httpd.conf
    3) Scroll down to where it reads "Timeout 300"
    4) Change it to "Timeout 5"
    5) Scroll down to "MaxClients 150"
    6) Change it to "MaxClients 500"
    7) Type "service httpd stop"
    8) Wait for around 10 seconds
    9) Type "service httpd start"

    (you may want to create a backup of your httpd.conf file first just in case you make a mistake)

    Now requests will time out after 5 seconds. That means if there is no traffic between the server and the visitor for 5 seconds the connection will time out. This will improve the server's load dramatically and the DDoS processes will probably not reach Apache's maxClients value anymore. The timeout setting can be a problem if customers are uploading files using Perl or PHP scripts but apart from that it shouldn't cause any issues (i.e. it will not interrupt downloads) and so far no customer has complained.

    Now that Apache is working again you can try to find a way to lock the attacking IPs out which can be very tricky and you may be better off waiting until the attack ends.
     
    #3 driverC, Sep 27, 2006
    Last edited: Sep 27, 2006
  4. NightStorm

    NightStorm Well-Known Member

    Joined:
    Jul 28, 2003
    Messages:
    286
    Likes Received:
    4
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    Wouldn't this type of attack (Where zombies connect dozens of times to a single site and overload httpd) that mod_evasive was designed to block?
     
  5. driverC

    driverC Well-Known Member

    Joined:
    Jul 23, 2004
    Messages:
    112
    Likes Received:
    0
    Trophy Points:
    16
    No ! Not if the attack is comming from various different IPs where each IP does only access the server once every couple of seconds. And my experience is that such attacks do never come from a single IP.

    Enter the following command to find out if all accesses are comming from the same IP:

    netstat -na |grep :80 |sort

    Note: If you see the same IP multiple times (i.e. 10 times) it doesn't mean that it is attacking your server.
     
    #5 driverC, Sep 27, 2006
    Last edited: Sep 27, 2006
  6. student

    student Active Member

    Joined:
    Aug 9, 2006
    Messages:
    38
    Likes Received:
    0
    Trophy Points:
    6
    Hi,
    driverC, great thanks for help, after applying your steps, this kind of DoS has been stopped :)

    Speaking about strange requests, also i see (quite often) lots of connections (in Apache Status) such as:
    136-14 | - | 0/0/45 | . | 0.01 | 33780 | 1 | 0.0 | 0.00 | 0.90 | 80.182.50.66 | (unavailable) | GET /img/Biedrona.gif HTTP/1.1

    as you can see, te 'Vhost' is '(unavailable)'. These connections sometimes are existing for a few hours and comming from single IP (server is not dropping them), is it normal? Should i consider this as some kind of DoS attempt?
     
  7. driverC

    driverC Well-Known Member

    Joined:
    Jul 23, 2004
    Messages:
    112
    Likes Received:
    0
    Trophy Points:
    16
    This may be a keep alive request or something. You could disable keep alive in httpd.conf (but there is no need to do that unless your servers are overloaded). I am not sure why it says unavailable in the VHost section. Maybe it is an access to your server's IP (i.e. someone accessing http://IP/~username/index.htm. You can disable that in Cpanel's security settings (you should really disable this for security reasons and because it allows people to steal bandwidth from other users).
     
  8. jols

    jols Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,111
    Likes Received:
    2
    Trophy Points:
    38
    I would love to disable this, but then how would our newly hosted customers be able to see their web pages before their domain becomes active at the server?
     
  9. dexus

    dexus Well-Known Member

    Joined:
    Jan 14, 2006
    Messages:
    169
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    No, that is not the case, I was testing that strenge thing on a server that have only two domains and I always have such 'Vhost' is '(unavailable)' connections to existing files on those domains. When I trie to access through IP adress, like you suggested, that is always shown as:

    (unavailable) | GET /~user/path/file.php

    end this "strange" connections are always shown as:

    (unavailable) | GET /path/file.php

    as you notice without "~" and without username, just path on the unavailable host, and that just doesn't have any sense.

    Does anyone have some logical explanation for this. I am just lost here.
     
  10. Spiral

    Spiral BANNED

    Joined:
    Jun 24, 2005
    Messages:
    2,023
    Likes Received:
    7
    Trophy Points:
    0
    That is ridiculously easy !!!!

    Just setup wildcard DNS and assign all new accounts a temporary subdomain address!

    We have mod_userdir disabled in all our servers (actually not even compiled) so
    it is not even possible to use and "/~username" at all whatsoever!

    In place of that though, we have a global server domain setup and through
    the use of wildcard DNS and a one line brainless easy modification to Cpanel's apache
    templates in /usr/local/cpanel/etc/httptemplates, any new accounts that we create
    through WHM or the Cpanel API automatically have a temporary subdomain setup.

    Users would have something like: http://login.domain.com that they can
    use until their own domain finishes transferring.

    How?

    Let's say your domain used for user subdomains is "mydomain.com" and
    your shared IP address for your server is "1.2.3.4" then you would do this ....

    1. Login to your server via SSH as root

    2. Edit /var/named/mydomain.com.db and add the following line:
    Code:
    * IN A 1.2.3.4
    
    3. Restart your DNS server
    Code:
    /etc/rc.d/init.d/named restart
    
    4. Edit /usr/local/cpanel/httptemplates/apache1/standard ('apache2' if you use apache2)

    Under the ServerAlias lines in the template, add a new line:
    Code:
    ServerAlias %user%.mydomain.com
    
    5. Restart your apache server
    Code:
    /usr/local/apache/bin/apachectl restart
    
    Existing accounts created before this change won't be effected but any new accounts
    you create would automatically have an alternate subdomain address of
    http://login.mydomain.com where "login" is the user's login name.
     
    #10 Spiral, Mar 12, 2007
    Last edited: Mar 12, 2007
  11. ramprage

    ramprage Well-Known Member

    Joined:
    Jul 21, 2002
    Messages:
    667
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Canada
    That's not exactly good advice..... you increased the allowed number of connections which in a real DDoS attack will end up crashing yoru server becaue the hardware can't handle Apache with that many connections going.

    Instead raise your MaxClients value slightly. You're using the default value anyways, try something like 200.

    You should use a DDoS tool combined with your firewall as well. This will help block IPs who are connecting too much from the usual automatically. I've seen attacks where some IPs are connected over 4,000 times (per single IP)

    Viewing Apache status from shell:

    lynx localhost/whm-server-status
     
  12. Arsenico

    Arsenico Member

    Joined:
    Apr 27, 2006
    Messages:
    24
    Likes Received:
    0
    Trophy Points:
    1
    Try limiting the connections that one ip address can make on the port 80

    with IPFW:
    ipfw add allow tcp from any to me dst-port 80 in limit src-addr 15


    Where 15 is the max of connections that an ip addres can create.

    also try with this.

    ipfw add 65534 allow tcp from any to any limit src-addr 50


    Where 50 is the max of connections that 1 ip address can make to the server.

    You can change it. (dont know how to do it on IPTABLES, sorry :(, im married with UNIX and ipfw :P)


    also in the HTTPD.CONF

    RLimitMEM
    RLimitCPU

    http://www.apacheref.com/ref/http_core/RLimitCPU.html
    http://www.apacheref.com/ref/http_core/RLimitMEM.html


    u'll need to set those to not get overloaded all the time.
     
  13. CaMer0n

    CaMer0n Well-Known Member

    Joined:
    Nov 8, 2004
    Messages:
    59
    Likes Received:
    0
    Trophy Points:
    6
    I'd just like to thank driverC for his http.conf configuration.
    I was getting massive CPU usage from Apache, mostly due to Yahoo and MSN bots it seems.
    Every process was using 10-30% of the CPU.
    After making the modifications below, the problem has vanished.

    Thanks again
    Cameron

    Code:
    1) SSH into your server
    2) Enter this command: pico /usr/local/apache/conf/httpd.conf
    3) Scroll down to where it reads "Timeout 300"
    4) Change it to "Timeout 5"
    5) Scroll down to "MaxClients 150"
    6) Change it to "MaxClients 500"
    7) Type "service httpd stop"
    8) Wait for around 10 seconds
    9) Type "service httpd start"
    
     
Loading...

Share This Page