The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

One popular domain brings down whole server ?

Discussion in 'General Discussion' started by 4u123, Feb 23, 2006.

  1. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    One site receiving a large amount of traffic cauesd a high load on a server yesterday. I didnt think much of it as the server was pretty full anyway.

    I moved this domain onto another server with only about 20 accounts and its become overloaded within 10 minutes of propagation completing.
    Is it really possible for one site to be so popular that a normal server (intel 2.8 512mg ram) cant cope with it ?

    Could this be some kind of http based attack ?

    Everything else is normal on the server, nothing strange in /tmp - just a large amount of http to this one domain.

    Server restarted - removed the account, all back to normal - except that I cant put that customer onto any of our servers withoout major load problems.

    Checking apache status even now shows lots of requests for images from that site from many different locations.

    In 3 years of using cpanel, ive not come across this before.

    I'd be very grateful for any advice.
     
    #1 4u123, Feb 23, 2006
    Last edited: Feb 23, 2006
  2. WestBend

    WestBend Well-Known Member

    Joined:
    Oct 12, 2003
    Messages:
    173
    Likes Received:
    0
    Trophy Points:
    16
    very easily...

    however you are not giving any details whatsoever on the nature of the site or the images.

    A server can choke if the eth0 is maxed. It could be choked due to swap and so on and so on..
     
  3. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    Its a site dedicated to cartoons which contains a fairly large number of screenshot type images and some video. Looking at the stats, there have been over 6 million image hits this month and about 40,000 unique visitors.
     
  4. WestBend

    WestBend Well-Known Member

    Joined:
    Oct 12, 2003
    Messages:
    173
    Likes Received:
    0
    Trophy Points:
    16
    Are you preventing hotlinking?
    Have you looked at your traffic per second and seen if you have maxed out your network card?

    Is it static html or database driven?

    Have you tailed the access log to see if access is directly to the images or pages being called as well?

    Whats the output from top?
     
  5. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    No, this issue only started happening yesterday - the customer doesnt have hotlinking enabled.

    I wouldnt know how to do that.

    PHP but no db

    Access log looks pretty normal, pages and images being requested.

    As mentioned in my previous post - the server was overloaded - Ive been forced to remove the domain from the httpd.conf to stop it being accessed.
     
  6. MMarko

    MMarko Well-Known Member

    Joined:
    Apr 18, 2005
    Messages:
    316
    Likes Received:
    0
    Trophy Points:
    16
    We have couple accounts that submit adult galleries - each account is getting about 50-60,000 uniques daily (there is 3-4 accounts with that load) and server load is 0.1 - 0.6

    You have to optimize apache, php and mysql. You have to remove large rewrite rules in htaccess and large rules in mod_sec.
     
  7. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    It could be the modsec config I guess but I'm not really prepared to change that just for one site. As it stands we are only using the rules file and exclude file from gotroot.
     
  8. elliotcooper

    elliotcooper Well-Known Member
    PartnerNOC

    Joined:
    May 18, 2005
    Messages:
    56
    Likes Received:
    0
    Trophy Points:
    6
    In order to check your bandwidth in real time you should use a package called iptraf. If you ssh into the machine and use your package manager like yum to install it. Once installed you just type:

    iptraf

    Select the 'General Interface Statistics' option. This will give you a reading of exactly how much data your server is using at that moment which is updated in real time. There are also other useful options in iptraf for monitoring your server's traffic.

    If you are not maxing your connection then the problem with apache is usually too many connections being created. If you have allowed apache to open too many similtanious connections then it will consume all your RAM and put the machine into swap wich will cause load problems and bring it to a grinding halt pdq. To check this you need to see how many connections you have allowed apache. If you run this command:

    less /etc/httpd/conf/httpd.conf | grep MaxClients

    As a rule of thumb you should count on each process needing 4MB of RAM. So if you have 1GB of memory then you should set this to something like 250. This is pretty conservative but it means that you should keep out of swap.

    You should ckeck to see if you are hitting the max clients limit by running:

    ps auxf | grep httpd | wc -l

    If this number is the same or really clost to the max clients then you will need to increase the number as long as you have the resources to support it. You can check if it causes swapping with the instructions listed below

    If you need more apache process then apache can provide without swapping then you need to look at installing a much more light weight http server like thttpd to serve the static content like images. What you can do set thttpd to listen on a different IP address. Then set an A record for something like img.domain.com to this IP address. You then update all your html to grab the images using that host name. This will leave apache to handle the dynamic stuff and a super fast resource light http server like thttpd to handle the static stuff. I took the load of a really busy forum server from around 250 to 0.3 usng this method.

    You can find the docs for thttpd here:

    http://www.acme.com/software/thttpd/

    You can check if you machine is swapping by first of all running this command:

    free -m

    This will tell you how much swap your machine is using. Having some swap is pretty normal as linux will dump data there that is not bein activly accessed by the os. What is bad is when the os needs to use the swap as an extention of RAM to store active data. To check this you need to run:

    vmstat 5

    This will show you an how much data is writtten and read to swap every 5 seconds. Ideally, you should see a lot of zeros in the swap coloum. If you are seeing a lot of figures in the swap colum turn down the number of apache processes restart apache and check again.
     
    #8 elliotcooper, Feb 24, 2006
    Last edited: Feb 24, 2006
  9. 4u123

    4u123 Well-Known Member
    PartnerNOC

    Joined:
    Jan 2, 2006
    Messages:
    765
    Likes Received:
    1
    Trophy Points:
    18
    Thanks.

    I looked into the mod security config and it looks like that was the problem.
     
Loading...

Share This Page