How to prevent scrapers using multiple IPs

[greektranslator]

At times I get massive spikes of "visitors" coming from very wide IP ranges. For example:
xxx.yyy.145.193
xxx.yyy.159.123

Where xxx.yyy are the same numbers. Would manual banning such huge ranges be meaningful? Is there any other easier way to counteract them? I do have CSF installed.

 

[cPRex]

Hey there! Do you specifically know if they are bots from a certain provider? If so, you could setup a robots.txt to keep them from indexing your content, or you could just block the entire IP range. There's nothing wrong with blocking a whole range, but it does open up the possibility of you blocking user's IPs that were not intended.

 

[greektranslator]

"but it does open up the possibility of you blocking user's IPs that were not intended" - yes, that is what I was worried about. Also, such a huge range blocking would affect server performance?

 

[cPRex]

CSF lets you use CIDR notation so you don't have a massive file with all the IPs listed, so that saves on resources significantly.

 

[keat63]

Perform a quick google search on the IP's, chances are xxx.yyy are from the same country and you may find that you have no dealings with such country.
In which case just block the whole country.
Problem solved.
I have half a dozen parts of the globe blocked.

 

[greektranslator]

CSF lets you use CIDR notation so you don't have a massive file with all the IPs listed, so that saves on resources significantly.

Still, the system has to check for numerous IPs nonetheless or using the ranges implies that this does not happen?

Perform a quick google search on the IP's, chances are xxx.yyy are from the same country and you may find that you have no dealings with such country.
In which case just block the whole country.
Problem solved.
I have half a dozen parts of the globe blocked.

Yeah, the source was Singapore Still, someone could launch the attack via a proxy service.

The point is, I am looking for an automated solution that will recognize irregular visit patterns from a group of similar IPs. Is there something like this?

 

[cPRex]

You'd have to contact CSF to see how they handle the CIDR ranges internally - I'm not sure what processing goes into that on their end. While this is an older post, a few users commented that they have blocked many ranges with no performance issues:

Blocking a range of IPs in CSF?

Hi Seeking opinions on the advisability of blocking some countries by range on a webserver in the csf deny. The server itself is high-powered with lots of resources. We are setting up a new server specifically for domestic sites. We have config server installed, and have some tight rules in...

forums.cpanel.net

It sounds like you're looking less at a firewall solution and more like an external filtering solution with more advanced tools. You may want to speak with your host to see if they offer anything like that with the ability to intercept traffic before it reaches your machine.

 

[keat63]

Personally, I feel that these things go in waves.
Block Singapore, forget about.
Unblock Singapore in 3 months time and you'll probably find that it's gone away.

 

[vicycletwo]

Rather than blacklisting bots, maybe you should whitelist them. If you don't want to kill your search results for the top few engines, you can whitelist their user-agent strings, which are generally well-publicized. The less ethical bots tend to forge user-agent strings of popular web browsers. The top few search engines should be driving upwards of 95% of your traffic.