Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

CSF and Google bots ???

Discussion in 'Security' started by furquan, May 8, 2012.

  1. furquan

    furquan Well-Known Member

    Joined:
    Jul 27, 2002
    Messages:
    471
    Likes Received:
    4
    Trophy Points:
    168
    Hello Every one,

    Since i am using CSF on my server, my clients are complaining that their google rankings are coming down as Google cannot communicate to the domains on the server and this is what it has to say :-

    URLs timed out - Network availability of your DNS and web servers

    I need to know how can i white list Google bots in CSF so that this issue gets addressed ??

    This is what Google has to say :

    Google doesn't post a public list of IP addresses for webmasters to whitelist. This is because these IP address ranges can change, causing problems for any webmasters who have hard coded them. The best way to identify accesses by Googlebot is to use the user-agent (Googlebot).


    Hence, How can we add Googlebot to CSF ?

    Please assist !!!!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  2. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    16,517
    Likes Received:
    425
    Trophy Points:
    583
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    You might do better to find out why google is being blocked in the firewall and solve that. For example, is it because of too many file not found?

    As one example of many I'm sure. Check your blocked list and see if you can make out why google is being blocked from the text in the blocked file.

    White listing an IP should be used very sparingly, if at all, for anyone other than the Server Administrator, IMHO.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. -GR-

    -GR- Active Member

    Joined:
    May 2, 2012
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    56
    cPanel Access Level:
    Root Administrator
    At the CSF plugin in WHM you can view the latest log entries by how ever many line numbers you enter and look for any entries that have to do with htaccess.
     
  4. mtindor

    mtindor Well-Known Member

    Joined:
    Sep 14, 2004
    Messages:
    1,346
    Likes Received:
    59
    Trophy Points:
    178
    Location:
    inside a catfish
    cPanel Access Level:
    Root Administrator
    A lot of Google crawling happens from IP addresses in the 66.249.64.0 - 66.249.91.255 IP space. There might be other Google blocks used for crawling as well. Check your firewall to see if anything in that space is listed as being blocked.

    Code:
    iptables -L -n -v|grep '66\.249\.'
    If you see any 66.249.x.x adddresses listed there, see where they are listed in csf files:

    Code:
    grep '66\.249\.'   /etc/csf/*
    Then you can get an idea what mechanisms in CSF / LFD are triggering the block, if it is indeed CSF / LFD blocking anything.

    Mike
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice