Rewrite Rule & Robot.txt files

[serina]

How do i block googlebot and other search engines form accessing pages that have Rewrite Rule. An example

.htaccess :
-----------
RewriteRule ^shop\-locale\-(.*).html /cgi-bin/mymall/shop.cgi?blah=$1 [L]

robot.txt
--------
User-agent: *
Disallow: /shop-*.html$
Disallow: shop-*.html$

Is the robot.txt file correct? Robots are still accessing the site causing it to be disabled by the systems administrator. MSN and Wisenutbot are still accessing those scripts. I added the last line "Disallow: shop-*.html$". Is there anything else I have to do?

thanks in advance!!
--serina

[stevel]

First, it is supposed to be robots.txt with an s.

Second, this is not a regexp. It is a match of the initial part of the URL. So what you would probably want is:

Disallow: /shop

This would block robots from visiting any page starting with /shop.

[serina]

Thank you!!