Robots.txt in error log

[eBay33]

I am continually receiving a robots.txt file not found error message in my error log. Any ideas on where this is coming from and who is trying to access this file that does not exist? Thanks for any help! Brian

[Bud]

Search engines use bots to crawl the net and assemble lists of urls to incorporate into the search engine results.

When a bot comes to your site, if it is a nice bot, it will ask for the robots.txt file. This file tells the bot which pages on your site, if any, it can include in it's list.

The robots.txt file is something that you can create to keep certain pages/directories from being indexed into search engines.

Most legit bots will obey the robots.txt, others like email harvesters tend to ignore it.

~bud