The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Unknown robot (identified by 'bot*')

Discussion in 'General Discussion' started by shacker23, Oct 21, 2014.

  1. shacker23

    shacker23 Well-Known Member

    Joined:
    Feb 20, 2005
    Messages:
    263
    Likes Received:
    1
    Trophy Points:
    16
    A lot of my customer sites are seeing HUGE upticks in traffic due to unidentified bots. As in, 90% of traffic coming from unidentified bots. awstats reports them as

    Code:
    Unknown robot (identified by 'bot*')
    There has always been quite a bit of this, but lately it seems far worse. I'm starting to think it might make sense to set up server-wide rules to whitelist the major/important bots and drop the rest. Has anyone tried this? Have you ended up accidentally over-blocking good bots? I don't want to get into an endless cat-and-mouse game - just looking for a good formula that will let as much good bot traffic through as possible while dumping the rest. If you've done, did you do it at the apache level or firewall level?

    Any good tools or formulae to share?

    Thanks.
     
  2. shacker23

    shacker23 Well-Known Member

    Joined:
    Feb 20, 2005
    Messages:
    263
    Likes Received:
    1
    Trophy Points:
    16
    cPanel's new ModSecurity module landed a few days ago and I've been giving it a workout. I added and activated a single custom rule, which culls bad bot activity by cross referencing a bots RBL from sbl-xbl.spamhaus.org. In the past eight hours it's blocked ~50,000 requests by bad bots to all of my customer sites. Absolutely incredible.

    The hits are not just on comment forms etc., but on images, homepages, RSS feeds, PDFs, everything. Will let it run for a couple of days then use awstats to see how this affects bandwidth usage by the heaviest-hit customers.

    Here's the rule I'm using (all on one line of course):

    SecRule REMOTE_ADDR "@rbl sbl-xbl.spamhaus.org" "phase:1,id:'981138',t:none,pass,nolog,auditlog,msg:'RBL Match for SPAM Source',tag:'AUTOMATION/MALICIOUS',severity:'2',setvar:'tx.msg=%{rule.msg}',setvar:tx.automation_score=+%{tx.warning_anomaly_score},setvar:tx.anomaly_score=+%{tx.warning_anomaly_score},setvar:tx.%{rule.id}-AUTOMATION/MALICIOUS-%{matched_var_name}=%{matched_var},setvar:ip.spammer=1,expirevar:ip.spammer=86400,setvar:ip.previous_rbl_check=1,expirevar:ip.previous_rbl_check=86400,skipAfter:END_RBL_CHECK"
     
  3. ITGabs

    ITGabs Well-Known Member

    Joined:
    Jul 30, 2013
    Messages:
    81
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    I had a problem with bots but not like 90%

    I am checking your script shacker23

    Thanks!
     
  4. shacker23

    shacker23 Well-Known Member

    Joined:
    Feb 20, 2005
    Messages:
    263
    Likes Received:
    1
    Trophy Points:
    16
    I didn't write the rule - it's directly out of the spiderlabs modsec rules manual on github. But yeah - I'm seeing overall traffic cut down on most large customer sites by 50% overnight. No false positives to report (yet). Very promising.
     
  5. shacker23

    shacker23 Well-Known Member

    Joined:
    Feb 20, 2005
    Messages:
    263
    Likes Received:
    1
    Trophy Points:
    16
    OK, I misspoke on the traffic reduction - looks like awstats hadn't completed its run when I wrote earlier. I am seeing the massive numbers of log entries in ModSecurity, but I am NOT seeing a huge reduction in traffic after all via awstats.

    My understanding is that ModSecurity hits should be appearing in apache's error_log, but I'm not seeing them there. It feels like mod security is logging, but not actually blocking. Is there some integration option or documentation I'm not seeing somewhere?

    Thanks.
     
  6. smoge

    smoge Well-Known Member

    Joined:
    Jul 2, 2004
    Messages:
    52
    Likes Received:
    0
    Trophy Points:
    6
    Do you have CSF (Config Server Firewall) installed? It will block your mod_security hits.
     
  7. shacker23

    shacker23 Well-Known Member

    Joined:
    Feb 20, 2005
    Messages:
    263
    Likes Received:
    1
    Trophy Points:
    16
    Oh! Surely there must be a way to configure ConfigServer to work *with* ModSecurity rather than against it? Any idea how?
     
  8. shacker23

    shacker23 Well-Known Member

    Joined:
    Feb 20, 2005
    Messages:
    263
    Likes Received:
    1
    Trophy Points:
    16
    OK, so cpanel's modsecurity logs to /usr/local/apache/logs/modsec_audit.log . In CSF configuration, I set MODSEC_LOG to that path and restarted CSF.

    So then I tail -f /var/log/lfd.log . I see CSF's normal block entries showing up in lfd.log, but no sign of the tons of hits that are showing up in the ModSecurity log watcher in the gui.

    Have I got things wired up wrong, or am I looking in the wrong place?
     
  9. reggaebkk

    reggaebkk Registered

    Joined:
    Nov 4, 2014
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Hi,
    I was wondering if it is feasible to write a rule that will allow all known bots except a list of known bots that I can update in a file; and disallow all unknown bots except a list of allowed unknown bots that I can update in a 2nd file
    When I mean known and unknown bots, I refer to what I see in awstats, it seems awstats can identify most mainstream bots...
    Is such a rule possible? Advisable?
     
Loading...

Share This Page