Question about .htaccess

Tien · 1-27-03, 09:10 AM

It i use .htaccess to disable user browse the directory, will that disable the robots spider too?

rainbore · 1-27-03, 09:19 AM

No. Search engines only follow links on webpages. They don't browse directories looking for pages. As long as you provide a link to a page, the search engines will find it. The only real exception is for pages and directories that you take steps to keep them from crawling though a robots.txt file installed in your main directory, or though <meta> tags on individual pages. The robots.txt file is by far the more effective method for controlling where search engines travel through your site.

Good luck!

Tien · 1-27-03, 09:21 AM

Thanks

1-27-03, 09:10 AM	#1
Tien Join Date: May 2002 Location: MA Posts: 488 Reputation: 9	Question about .htaccess It i use .htaccess to disable user browse the directory, will that disable the robots spider too?

1-27-03, 09:19 AM	#2
rainbore Rick Join Date: May 2002 Location: Minneapolis, MN Posts: 1,696 Reputation: 280	No. Search engines only follow links on webpages. They don't browse directories looking for pages. As long as you provide a link to a page, the search engines will find it. The only real exception is for pages and directories that you take steps to keep them from crawling though a robots.txt file installed in your main directory, or though <meta> tags on individual pages. The robots.txt file is by far the more effective method for controlling where search engines travel through your site. Good luck! __________________ Rick Trethewey

1-27-03, 09:21 AM	#3
Tien Join Date: May 2002 Location: MA Posts: 488 Reputation: 9	Thanks