MarketPosition.com Archives for Oct, 2000
October 15, 2000
How and Why to Build a Robots.txt
Some of you have asked "How do I keep 'search engine A' from indexing pages designed for 'search engine B'." The answer is to use a robots.txt file. There are also other reasons for wanting to keep search engines from indexing some or all pages on a site. Therefore, I’ve put together this detailed article to show you how to do that, and to avoid common mistakes that are made all too often.
