The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Requesting a Deny Agent function

Discussion in 'General Discussion' started by lottocheatah, May 17, 2011.

Thread Status:
Not open for further replies.
  1. lottocheatah

    lottocheatah Active Member

    Joined:
    May 17, 2011
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    In some instances the legacy latest visitors log entry will identify the agent software being used as hacking sofware. In others the agent field is deliberately undeclared. With WIFI and dynamic I.P. numbers the I.P. Deny Manager has become substantially ineffective at denying this kind of traffic. It would be nice to be able to "Deny Agent" based on the content or lack of content in the Agent field of the Legacy Latest Visitors Report.
     
  2. cPanelDavidG

    cPanelDavidG Technical Product Specialist

    Joined:
    Nov 29, 2006
    Messages:
    11,279
    Likes Received:
    8
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    So, basically you want a user interface to easily manage the ability to deny based on agent, a capability currently available by means of manually editing a .htaccess file?

    In the interim, if you are using site software such as WordPress, there are already plugins available to block "bad agents," some of which automatically update their list of "bad agents."
     
  3. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,452
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Can't Modsecurity can handle this?
     
  4. lottocheatah

    lottocheatah Active Member

    Joined:
    May 17, 2011
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    I'm don't use wordpress. I use the features in cPanel. I'm a website user. I don't know anything about editing system files if the function is not available on the cPanel. The I.P. Deny Manager has become quite ineffective at keeping out bad actors, but if I can I.D. a bad agent to an Agent Deny function then that agent can be blocked regardless of the I.P. number they are using.

    I.P. Deny has become obsolete. The deny relationship should refer to the agent, not the I.P.
     
  5. lottocheatah

    lottocheatah Active Member

    Joined:
    May 17, 2011
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    I am a user of the cPanel. I don't find anything on the cPanel entitled "Modsecurity"
     
  6. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,452
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    You'll need to speak with your host to find out if Modsecurity is enabled on your server. There are rulesets to block known bad agents.

    You can also block via .htaccess. Lots on google about this.

    Keep in mind, you can fake your User Agent string.
     
  7. lottocheatah

    lottocheatah Active Member

    Joined:
    May 17, 2011
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
     
  8. cPanelDavidG

    cPanelDavidG Technical Product Specialist

    Joined:
    Nov 29, 2006
    Messages:
    11,279
    Likes Received:
    8
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    cPanel&WHM supports .htaccess files, we just do not provide a facility by which to easily block user agents at this time. However, the capability is already present and in use by those using cPanel.
     
  9. lottocheatah

    lottocheatah Active Member

    Joined:
    May 17, 2011
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    I will check the .htaccess file details and see if this suits the purpose.
     
  10. lottocheatah

    lottocheatah Active Member

    Joined:
    May 17, 2011
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    I found a bit of code for this file but I cannot file the existing file in any website folder. Is this an optional file? Is there a generic one available? Where does it mount?
     
  11. lottocheatah

    lottocheatah Active Member

    Joined:
    May 17, 2011
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    It is starting to get all quite confusing for someone who is just a cPanel user and not a java programmer. What is the prefix for the .htaccess file? I am finding info that suggests I need an exact string match for the bad agent and that I need an additional file "ModRewrite" to match to a sub-string portion of the Agent field. Is this true?

    All this said, why is the Agent Deny function not on the cPanel?
     
  12. cPanelDavidG

    cPanelDavidG Technical Product Specialist

    Joined:
    Nov 29, 2006
    Messages:
    11,279
    Likes Received:
    8
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    This is an optional file. If you do not see this in File Manager, first try to create a .htaccess file.

    If you still do not see the file after creating it, keep in mind that .htaccess starts with a period so this makes it a "hidden" file on *nix environments. You will need to go to the bottom of the cPanel home screen and reset all preferences, then click on File Manager and in the popup that appears, ensure you can see hidden files.

    .htaccess is most often stored in your public_html directory.
     
  13. cPanelDavidG

    cPanelDavidG Technical Product Specialist

    Joined:
    Nov 29, 2006
    Messages:
    11,279
    Likes Received:
    8
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    Keep in mind that .htaccess is an extremely commonly used tool. I would recommend looking for tools via an internet search that can generate these strings for you. When it comes to .htaccess files, there are utilities to generate content for .htaccess files for nearly anything you can think of.
     
  14. lottocheatah

    lottocheatah Active Member

    Joined:
    May 17, 2011
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    ".htaccess" is a just the file extension. What is the prefix, ie; what is the entire filename of the .htaccess file?
     
  15. cPanelDavidG

    cPanelDavidG Technical Product Specialist

    Joined:
    Nov 29, 2006
    Messages:
    11,279
    Likes Received:
    8
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    The entire filename of .htaccess is .htaccess - it is also not a file extension, it is a hidden file (since it starts with .).

    Remember, in *nix, text following a dot does not have the same meaning it may on Windows.

    This thread has derailed far outside of being a feature request, I'll be moving this thread to a more appropriate forum.
     
  16. lottocheatah

    lottocheatah Active Member

    Joined:
    May 17, 2011
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    Why move the thread? This is part of the dialog for my request for a Deny Agent function on the cPanel. It should be obvious if you suggest sending a cPanel user, ie; not a programmer or an Apachie tech on route that suggests he starts plugging things into his website environment he knows nothing about rather than address the request that you are derailing the request. Please, this feature is needed. There is no security for users of the cPanel without it.
     
  17. cPanelDavidG

    cPanelDavidG Technical Product Specialist

    Joined:
    Nov 29, 2006
    Messages:
    11,279
    Likes Received:
    8
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    Normally, I would just copy a post from a thread to maintain the feature request. However this thread has mostly been focused on technical assistance. If you want to create a new feature request thread strictly as a feature request and reference this thread in that feature request, then use only this thread for technical assistance - I think that would be mutually amicable.
     
  18. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,452
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    I should have been paying closer attention here, my apologies.

    For a quick and easy way to block bad user agents on an end user level, why not go with a robots.txt file in the root of your domain?

    yourdomain.com/robots.txt

    For an example, open notepad on your computer and paste this into it.



    Code:
    User-agent: AboutUsBot
    Disallow: / 
    
    User-agent: BoardReader
    Disallow: / 
    
    User-agent: dotbot 
    Disallow: / 
    
    #User-agent: Googlebot 
    #Disallow: /
    
    #User-agent: *
    #Disallow: /
    
    User-agent: larbin
    Disallow: /
    
    User-agent: b2w/0.1
    Disallow: /
    
    User-agent: Copernic
    Disallow: /
    
    User-agent: psbot
    Disallow: /
    
    User-agent: Python-urllib
    Disallow: /
    
    User-agent: URL_Spider_Pro
    Disallow: /
    
    User-agent: CherryPicker
    Disallow: /
    
    User-agent: EmailCollector
    Disallow: /
    
    User-agent: EmailSiphon
    Disallow: /
    
    User-agent: WebBandit
    Disallow: /
    
    User-agent: EmailWolf
    Disallow: /
    
    User-agent: EuripBot/1.0
    Disallow: /
    
    User-agent: ExtractorPro
    Disallow: /
    
    User-agent: CopyRightCheck
    Disallow: /
    
    User-agent: Crescent
    Disallow: /
    
    User-agent: SiteSnagger
    Disallow: /
    
    User-agent: ProWebWalker
    Disallow: /
    
    User-agent: CheeseBot
    Disallow: /
    
    User-agent: LNSpiderguy
    Disallow: /
    
    User-agent: Alexibot
    Disallow: /
    
    User-agent: Teleport
    Disallow: /
    
    User-agent: TeleportPro
    Disallow: /
    
    User-agent: MIIxpc
    Disallow: /
    
    User-agent: Telesoft
    Disallow: /
    
    User-agent: Website Quester
    Disallow: /
    
    User-agent: WebZip
    Disallow: /
    
    User-agent: moget/2.1
    Disallow: /
    
    User-agent: WebZip/4.0
    Disallow: /
    
    User-agent: WebStripper
    Disallow: /
    
    User-agent: WebSauger
    Disallow: /
    
    User-agent: WebCopier
    Disallow: /
    
    User-agent: NetAnts
    Disallow: /
    
    User-agent: Mister PiX
    Disallow: /
    
    User-agent: WebAuto
    Disallow: /
    
    User-agent: TheNomad
    Disallow: /
    
    User-agent: WWW-Collector-E
    Disallow: /
    
    User-agent: RMA
    Disallow: /
    
    User-agent: libWeb/clsHTTP
    Disallow: /
    
    User-agent: asterias
    Disallow: /
    
    User-agent: httplib
    Disallow: /
    
    User-agent: turingos
    Disallow: /
    
    User-agent: spanner
    Disallow: /
    
    User-agent: InfoNaviRobot
    Disallow: /
    
    User-agent: Harvest/1.5
    Disallow: /
    
    User-agent: Bullseye/1.0
    Disallow: /
    
    User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
    Disallow: /
    
    User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
    Disallow: /
    
    User-agent: CherryPickerSE/1.0
    Disallow: /
    
    User-agent: CherryPickerElite/1.0
    Disallow: /
    
    User-agent: WebBandit/3.50
    Disallow: /
    
    User-agent: NICErsPRO
    Disallow: /
    
    User-agent: Microsoft URL Control - 5.01.4511
    Disallow: /
    
    User-agent: Not Your Business!
    Disallow: /
    
    User-agent: Hidden-Referrer
    Disallow: /
    
    User-agent: DittoSpyder
    Disallow: /
    
    User-agent: Foobot
    Disallow: /
    
    User-agent: WebmasterWorldForumBot
    Disallow: /
    
    User-agent: spbot
    Disallow: /
    
    User-agent: SpankBot
    Disallow: /
    
    User-agent: Speedy Spider
    Disallow: /
    
    User-agent: BotALot
    Disallow: /
    
    User-agent: lwp-trivial/1.34
    Disallow: /
    
    User-agent: lwp-trivial
    Disallow: /
    
    User-agent: BunnySlippers
    Disallow: /
    
    User-agent: Microsoft URL Control - 6.00.8169
    Disallow: /
    
    User-agent: URLy Warning
    Disallow: /
    
    User-agent: Wget/1.6
    Disallow: /
    
    User-agent: Wget/1.5.3
    Disallow: /
    
    User-agent: Wget
    Disallow: /
    
    User-agent: LinkWalker
    Disallow: /
    
    User-agent: cosmos
    Disallow: /
    
    User-agent: moget
    Disallow: /
    
    User-agent: hloader
    Disallow: /
    
    User-agent: humanlinks
    Disallow: /
    
    User-agent: LinkextractorPro
    Disallow: /
    
    User-agent: Offline Explorer
    Disallow: /
    
    User-agent: Mata Hari
    Disallow: /
    
    User-agent: LexiBot
    Disallow: /
    
    User-agent: Web Image Collector
    Disallow: /
    
    User-agent: The Intraformant
    Disallow: /
    
    User-agent: True_Robot/1.0
    Disallow: /
    
    User-agent: True_Robot
    Disallow: /
    
    User-agent: BlowFish/1.0
    Disallow: /
    
    User-agent: JennyBot
    Disallow: /
    
    User-agent: MIIxpc/4.2
    Disallow: /
    
    User-agent: BuiltBotTough
    Disallow: /
    
    User-agent: ProPowerBot/2.14
    Disallow: /
    
    User-agent: BackDoorBot/1.0
    Disallow: /
    
    User-agent: toCrawl/UrlDispatcher
    Disallow: /
    
    User-agent: WebEnhancer
    Disallow: /
    
    User-agent: suzuran
    Disallow: /
    
    User-agent: TightTwatBot
    Disallow: /
    
    User-agent: TurnitinBot
    Disallow: /
    
    User-agent: VCI WebViewer VCI WebViewer Win32
    Disallow: /
    
    User-agent: VCI
    Disallow: /
    
    User-agent: Szukacz/1.4 
    Disallow: /
    
    User-agent: QueryN Metasearch
    Disallow: /
    
    User-agent: Openfind data gathere
    Disallow: /
    
    User-agent: Openfind 
    Disallow: /
    
    User-agent: Xenu's Link Sleuth 1.1c
    Disallow: /
    
    User-agent: Xenu's
    Disallow: /
    
    User-agent: Zeus
    Disallow: /
    
    User-agent: RepoMonkey Bait & Tackle/v1.01
    Disallow: /
    
    User-agent: RepoMonkey
    Disallow: /
    
    User-agent: Microsoft URL Control
    Disallow: /
    
    User-agent: Openbot
    Disallow: /
    
    User-agent: URL Control
    Disallow: /
    
    User-agent: Zeus Link Scout
    Disallow: /
    
    User-agent: Zeus 32297 Webster Pro V2.9 Win32
    Disallow: /
    
    User-agent: Webster Pro
    Disallow: /
    
    User-agent: EroCrawler
    Disallow: /
    
    User-agent: LinkScan/8.1a Unix
    Disallow: /
    
    User-agent: Keyword Density/0.9
    Disallow: /
    
    User-agent: Kenjin Spider
    Disallow: /
    
    User-agent: Iron33/1.0.2
    Disallow: /
    
    User-agent: Bookmark search tool
    Disallow: /
    
    User-agent: GetRight/4.2
    Disallow: /
    
    User-agent: FairAd Client
    Disallow: /
    
    User-agent: Gaisbot
    Disallow: /
    
    User-agent: Aqua_Products
    Disallow: /
    
    User-agent: Radiation Retriever 1.1
    Disallow: /
    
    User-agent: WebmasterWorld Extractor
    Disallow: /
    
    User-agent: Flaming AttackBot
    Disallow: /
    
    User-agent: Oracle Ultra Search
    Disallow: /
    
    User-agent: MSIECrawler
    Disallow: /
    
    User-agent: PerMan
    Disallow: /
    
    User-agent: searchpreview
    Disallow: / 
    
    User-agent: Baiduspider
    Disallow: / 
    
    User-agent: Yandex
    Disallow: / 
    
    User-agent: Yeti
    Disallow: /
    
    User-agent: Exabot
    Disallow: / 
    
    User-agent: YoudaoBot/1.0 
    Disallow: /
    
    User-agent: MLBot
    Disallow: / 
    
    
    Next, save that file named exactly this: robots.txt

    Next upload it via cPanel > File Manager, into the directory public_html

    More on this can be found here:

    The Web Robots Pages


    Blocking user agents might be useful in some circumstances, but this if used correctly is certainly only a small piece of a lot more security you should have going. Your host can explain more of this to you as to what sort of security they have on their server.

    Here's an article on using an .htaccess file thats a bit more advanced I think but you might like to see:
    How To Block Bad Bots and Spiders using .htaccess

    HTH!
     
  19. lottocheatah

    lottocheatah Active Member

    Joined:
    May 17, 2011
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    What will resetting all preferences do to other settings I have configured. Will it delete my Deny I.P. list?
     
  20. lottocheatah

    lottocheatah Active Member

    Joined:
    May 17, 2011
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    If I put in a robots.txt I will get no indexing at all! That is a terrible suggestion. I don't want to kill my indexing. I want to deny bad agents.
     
Loading...
Thread Status:
Not open for further replies.

Share This Page