Hammered by Googlebot , getting flooded by invalid URLs on one domain

Operating System & Version
Cloudlinux 7.8
cPanel & WHM Version
v88.0.11

hrodrigues

Registered
Jun 29, 2020
2
0
1
Brasil
cPanel Access Level
Reseller Owner
Hi all,

I have one particular customer who is getting flooded by Googlebot, with requests to inexistent URLs.

I have tried saying Google not to index my using meta tags, Disallow: on robots.txt, and on Google search console I requested clearing cache and temporary removal of the entire domain. None worked.

I still see hundreds of requests 24x7 on my apache status, as attached picture shows. some of them enjoying keepalive timer.

Blocking Googlebot ips on firewall work, but is NOT an option as this is a shared server, it would harm other server users by preventing their sites from getting crawled.

How can I improve that situation, perhaps a modsecurity rule denying Googlebot /24 subnet or it´s User Agent to that particular domain? If yes, What would the rule look like?

Please help
 

Attachments

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,304
1,252
313
Houston
Well you wouldn't want google to NOT index the site that would be bad for the site as a whole. I'd suggest looking at the suggestions from google for other users that have experienced issues with this like Change Googlebot crawl rate - Search Console Help
 
  • Like
Reactions: hrodrigues