PDA

View Full Version : Using robots.txt to disallow Google


preston
11-23-05, 03:19 PM
I've been getting slammed by traffic from an image on my website that was listed in google's image search.

I've removed the image and the content related to it, hoping that would help.

I've learned that I'm not able to use .htaccess to redirect traffic from images.google.com to another site in an effort to slow the bandwidth, etc.

What I'm trying to do now is put together a robots.txt file that will keep Google from crawling my site, which I'm hoping will take my site out of their search results.

Can anyone tell me if this will do it?

User-agent: Googlebot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

Also, does the robots.txt need to be placed inside the /htdocs directory or in the main root of the site in its parent directory?

Thanks for the help!
Preston

stevel
11-23-05, 03:59 PM
The file goes in htdocs. Yes, I think that file will do it, though if you want to discourage Google Images only, use just the second pair of lines.

preston
11-23-05, 04:32 PM
Thanks a bunch Steve. I hope the site gets crawled soon!

linnetwoods
11-23-05, 08:36 PM
Have you submitted your site to Google since you added that robots.txt? It might speed up their next visit...