How to prevent scrapers using multiple IPs

greektranslator

Well-Known Member
Jun 5, 2011
104
5
68
Greece
cPanel Access Level
Root Administrator
At times I get massive spikes of "visitors" coming from very wide IP ranges. For example:
xxx.yyy.145.193
xxx.yyy.159.123

Where xxx.yyy are the same numbers. Would manual banning such huge ranges be meaningful? Is there any other easier way to counteract them? I do have CSF installed.
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
6,923
912
313
cPanel Access Level
Root Administrator
Hey there! Do you specifically know if they are bots from a certain provider? If so, you could setup a robots.txt to keep them from indexing your content, or you could just block the entire IP range. There's nothing wrong with blocking a whole range, but it does open up the possibility of you blocking user's IPs that were not intended.
 

keat63

Well-Known Member
Nov 20, 2014
1,913
259
113
cPanel Access Level
Root Administrator
Perform a quick google search on the IP's, chances are xxx.yyy are from the same country and you may find that you have no dealings with such country.
In which case just block the whole country.
Problem solved.
I have half a dozen parts of the globe blocked.
 
  • Like
Reactions: cPRex

greektranslator

Well-Known Member
Jun 5, 2011
104
5
68
Greece
cPanel Access Level
Root Administrator
CSF lets you use CIDR notation so you don't have a massive file with all the IPs listed, so that saves on resources significantly.
Still, the system has to check for numerous IPs nonetheless or using the ranges implies that this does not happen?

Perform a quick google search on the IP's, chances are xxx.yyy are from the same country and you may find that you have no dealings with such country.
In which case just block the whole country.
Problem solved.
I have half a dozen parts of the globe blocked.
Yeah, the source was Singapore :) Still, someone could launch the attack via a proxy service.

The point is, I am looking for an automated solution that will recognize irregular visit patterns from a group of similar IPs. Is there something like this?
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
6,923
912
313
cPanel Access Level
Root Administrator
You'd have to contact CSF to see how they handle the CIDR ranges internally - I'm not sure what processing goes into that on their end. While this is an older post, a few users commented that they have blocked many ranges with no performance issues:


It sounds like you're looking less at a firewall solution and more like an external filtering solution with more advanced tools. You may want to speak with your host to see if they offer anything like that with the ability to intercept traffic before it reaches your machine.
 

keat63

Well-Known Member
Nov 20, 2014
1,913
259
113
cPanel Access Level
Root Administrator
Personally, I feel that these things go in waves.
Block Singapore, forget about.
Unblock Singapore in 3 months time and you'll probably find that it's gone away.
 
  • Like
Reactions: cPRex

vicycletwo

Registered
Jan 13, 2021
1
0
0
Nevada
cPanel Access Level
Website Owner
Rather than blacklisting bots, maybe you should whitelist them. If you don't want to kill your search results for the top few engines, you can whitelist their user-agent strings, which are generally well-publicized. The less ethical bots tend to forge user-agent strings of popular web browsers. The top few search engines should be driving upwards of 95% of your traffic.