The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Problems with Google Bot Overloading Server

Discussion in 'General Discussion' started by wrender, Mar 12, 2011.

  1. wrender

    wrender Well-Known Member

    Joined:
    Sep 29, 2007
    Messages:
    69
    Likes Received:
    3
    Trophy Points:
    8
    I am having an issue with a Google bot crawling a website on our server. Was wondering if anyone could shed some light. When I check our server load every once in a while sometimes it goes up to 8-10 (very high), and I see one of our accounts has multiple php processes running at 80-99% cpu load.

    When I check the domlogs, I noticed the following bot crawling the website (Below). When I block the bot's ip in the firewall the server load goes back down almost instantly to 0.25 server load which is normal for us.

    I have had two different ip addresses from Google bot that have been causing server overloads within the last few months. It has taken a lot of work to trace this back to this bot, and I can't figure out why it's happening.

    Here is a line of the bot in question.

    Code:
    66.249.72.243 - - [11/Mar/2011:22:04:12 -0700] "GET /events/calendar/0838-11-15 HTTP/1.1" 404 22273 "-" "                               Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    
     
  2. NaSRi

    NaSRi Member

    Joined:
    Aug 13, 2014
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Please up this issue , recently we have seen this
    Code:
    54.69.228.145 - - [04/Aug/2015:11:20:15 +0430] "POST /xmlrpc.php HTTP/1.0" 200 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;  http://www.google.com/bot.html)"
    54.69.3.69 - - [04/Aug/2015:11:20:15 +0430] "POST /xmlrpc.php HTTP/1.0" 200 633 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;  http://www.google.com/bot.html)"
    
    
    this made our server to be down . load was 250 !
     
    #2 NaSRi, Aug 4, 2015
    Last edited by a moderator: Aug 4, 2015
  3. Arkaic

    Arkaic Member

    Joined:
    Jun 23, 2015
    Messages:
    22
    Likes Received:
    3
    Trophy Points:
    3
    Location:
    United Kingdom
    cPanel Access Level:
    Root Administrator
    Block the IP.
     
Loading...

Share This Page