I am having an issue with a Google bot crawling a website on our server. Was wondering if anyone could shed some light. When I check our server load every once in a while sometimes it goes up to 8-10 (very high), and I see one of our accounts has multiple php processes running at 80-99% cpu load.
When I check the domlogs, I noticed the following bot crawling the website (Below). When I block the bot's ip in the firewall the server load goes back down almost instantly to 0.25 server load which is normal for us.
I have had two different ip addresses from Google bot that have been causing server overloads within the last few months. It has taken a lot of work to trace this back to this bot, and I can't figure out why it's happening.
Here is a line of the bot in question.
When I check the domlogs, I noticed the following bot crawling the website (Below). When I block the bot's ip in the firewall the server load goes back down almost instantly to 0.25 server load which is normal for us.
I have had two different ip addresses from Google bot that have been causing server overloads within the last few months. It has taken a lot of work to trace this back to this bot, and I can't figure out why it's happening.
Here is a line of the bot in question.
Code:
66.249.72.243 - - [11/Mar/2011:22:04:12 -0700] "GET /events/calendar/0838-11-15 HTTP/1.1" 404 22273 "-" " Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
