The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

CSF and Google bots ???

Discussion in 'Security' started by furquan, May 8, 2012.

  1. furquan

    furquan Well-Known Member

    Joined:
    Jul 27, 2002
    Messages:
    425
    Likes Received:
    0
    Trophy Points:
    16
    Hello Every one,

    Since i am using CSF on my server, my clients are complaining that their google rankings are coming down as Google cannot communicate to the domains on the server and this is what it has to say :-

    URLs timed out - Network availability of your DNS and web servers

    I need to know how can i white list Google bots in CSF so that this issue gets addressed ??

    This is what Google has to say :

    Google doesn't post a public list of IP addresses for webmasters to whitelist. This is because these IP address ranges can change, causing problems for any webmasters who have hard coded them. The best way to identify accesses by Googlebot is to use the user-agent (Googlebot).


    Hence, How can we add Googlebot to CSF ?

    Please assist !!!!
     
  2. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,480
    Likes Received:
    203
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    You might do better to find out why google is being blocked in the firewall and solve that. For example, is it because of too many file not found?

    As one example of many I'm sure. Check your blocked list and see if you can make out why google is being blocked from the text in the blocked file.

    White listing an IP should be used very sparingly, if at all, for anyone other than the Server Administrator, IMHO.
     
  3. -GR-

    -GR- Active Member

    Joined:
    May 2, 2012
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    At the CSF plugin in WHM you can view the latest log entries by how ever many line numbers you enter and look for any entries that have to do with htaccess.
     
  4. mtindor

    mtindor Well-Known Member

    Joined:
    Sep 14, 2004
    Messages:
    1,281
    Likes Received:
    37
    Trophy Points:
    48
    Location:
    inside a catfish
    cPanel Access Level:
    Root Administrator
    A lot of Google crawling happens from IP addresses in the 66.249.64.0 - 66.249.91.255 IP space. There might be other Google blocks used for crawling as well. Check your firewall to see if anything in that space is listed as being blocked.

    Code:
    iptables -L -n -v|grep '66\.249\.'
    If you see any 66.249.x.x adddresses listed there, see where they are listed in csf files:

    Code:
    grep '66\.249\.'   /etc/csf/*
    Then you can get an idea what mechanisms in CSF / LFD are triggering the block, if it is indeed CSF / LFD blocking anything.

    Mike
     
Loading...

Share This Page