How to disallow some crawler globally in httpd conf?

postcd

Well-Known Member
Oct 22, 2010
721
21
68
Hello,

if i want to disable some web crawler globally on all existing and future accounts, how to do it persistently?

example this can be added to .htaccess but i need persistent global file of httpd ?

I found Home » Service Configuration » Apache Configuration » Include Editor
there are 3 include files, not sure if i can use this and add mentioned thing? persistently?

Also what if i want to add some robots.txt rule globally?
like:
user-agent: AhrefsBot
Crawl-Delay: 3
 
Last edited:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,880
2,258
463