Apache Bug (Build 11.38.x)? (get /robots.txt)

[easyswiss]

We have a problem on 2 servers since ~july/august 2013, they have every day a lot of processes like this here:

945950 1/43/7079 G 1.80 68687 0 2.7 0.50 175.53 66.249.66.104 HOSTNAME:4 GET /robots.txt HTTP/1.1
872984 1/9/36597 G 0.19 11952 0 2.7 0.19 1167.38 66.249.66.36 HOSTNAME:4 GET /robots.txt HTTP/1.1

This results in heavy load (this processes take 100% cpu usage -> as nobody).

Is there a solution?
Add this IP's to the host access access control have no effect.
Only this "get /robots.txt processes need so much cpu.

 

[cPanelMichael]

Hello

Do you have a third-party firewall application such as CSF installed on your system? If not, it may be a good idea to install a firewall so you can block IP addresses from accessing your server. You may also want to install Mod_Security to help with certain access attempts.

Thank you.

 

[cPanelJamyn]

The 66.249.66.* address space belongs to Google's web crawlers. Blocking them will have a negative effect on your (or your customer's) website ranking, obviously. You can add a meta tag to the website if you really want google to stop visiting.

<meta name="googlebot" content="noindex">

You can also register for Google's Webmaster Tools, and change how fast (and how often) the googlebot crawls your website.