The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Googlebot crawls and crawls

Discussion in 'General Discussion' started by neonix, Aug 10, 2005.

  1. neonix

    neonix Well-Known Member

    Joined:
    Oct 21, 2004
    Messages:
    124
    Likes Received:
    2
    Trophy Points:
    0
    Googlebot has been crawling a few of my sites since yesterday. The sites are not huge. I banned one I.P (APF) for a few hours but upon restoring the i.p. the crawling restarted immediately.

    Is there any way to know if this is a genuine Googlebot or not. The i.p does show as it belongs to google but why is it on these sites for 2 days.....If I keep on banning google i.ps this will affect my customers...any suggestions upon the course of action...Thanks.


    66.249.65.232 - - [09/Aug/2005:04:29:17 +0530] "GET /index2.php?option=com_content&task=view&id=41&Itemid=59&pop=1&page=0 HTTP/1.1" 200 1514 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [09/Aug/2005:04:29:43 +0530] "GET /index.php?option=com_content&task=view&id=41&Itemid=3 HTTP/1.1" 200 15128 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [09/Aug/2005:04:30:16 +0530] "GET /news-feeds/index.php?option=com_content&task=view&id=13&Itemid=36 HTTP/1.1" 200 17661 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [09/Aug/2005:04:30:36 +0530] "GET /faqs/software-developement/what-documentation-does-fdsc-provide-.html HTTP/1.1" 200 17825 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [09/Aug/2005:04:31:33 +0530] "GET /news-feeds/index.php?option=com_registration&task=register HTTP/1.1" 200 15102 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [09/Aug/2005:04:31:57 +0530] "GET /index2.php?option=com_content&task=emailform&id=51 HTTP/1.1" 200 2259 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"


    66.249.65.232 - - [10/Aug/2005:03:26:29 +0530] "GET /FAQs/Software-Developement/index.php?option=com_content&task=view&id=41 HTTP/1.1" 200 14924 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:03:26:30 +0530] "GET /Software-Solutions/FAQs/Partners/ HTTP/1.1" 200 17879 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:03:26:33 +0530] "GET /Software-Solutions/index.php?option=com_content&task=view&id=41&Itemid=76 HTTP/1.1" 200 15165 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:03:26:34 +0530] "GET /index2.php?option=com_content&do_pdf=1&id=33 HTTP/1.1" 200 1689 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:03:26:34 +0530] "GET /FAQs/Partners/index.php?option=com_content&task=view&id=57&Itemid=74 HTTP/1.1" 200 15795 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:03:26:38 +0530] "GET /index2.php?option=com_content&do_pdf=1&id=17 HTTP/1.1" 200 2021 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:03:26:38 +0530] "GET /Partners/Value-Proposition/index.php?option=com_content&task=view&id=41&Itemid=48 HTTP/1.1" 200 15489 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

    66.249.65.232 - - [10/Aug/2005:21:20:39 +0530] "GET /faqs/index.php?option=com_content&task=view&id=41&Itemid=42 HTTP/1.1" 200 17061 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:21:20:48 +0530] "GET /home/index.php?option=com_weblinks&task=view&catid=27&id=13 HTTP/1.1" 302 5 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:21:21:05 +0530] "GET /partners/working-model/index.php?option=com_content&task=view&id=44&Itemid=68 HTTP/1.1" 200 15631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:21:21:12 +0530] "GET /links/Software-Solutions/Application-Form/Page_8.html HTTP/1.1" 200 17899 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:21:21:15 +0530] "GET /home/index.php?option=com_content&task=view&id=61&Itemid=56 HTTP/1.1" 200 16611 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:21:21:18 +0530] "GET /faqs/software-developement/FAQs/Partners/Books.htm HTTP/1.1" 200 18042 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:21:22:11 +0530] "GET /partners/value-proposition/index.php?option=com_content&task=view&id=61&Itemid=56 HTTP/1.1" 200 16760 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:21:22:21 +0530] "GET /registration/index.php?option=com_content&task=view&id=33&Itemid=55 HTTP/1.1" 200 17482 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.65.232 - - [10/Aug/2005:21:22:48 +0530] "GET /content/view/49/81/index.php?option=com_content&task=view&id=61&Itemid=50 HTTP/1.1" 200 15414 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
     
  2. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Did you go to the link in the UA tag?
    http://www.google.com/bot.html

    This explains about how it works and how to block it using a standard robots.txt file.
     
Loading...
Similar Threads - Googlebot crawls crawls
  1. keat63
    Replies:
    4
    Views:
    436

Share This Page