« How is WebPosition Going to Get Everyone in the Top 10? | How to Make Better Use of Images »
February 15, 1999
Solution to Disappearing Pages on Excite
I’ve reported in the past how Excite likes to cycle through every couple weeks and drop pages at random. A customer pointed out that the problem appears to be Excite’s limit of no more than 25 pages per Web site.
Apparently, every couple weeks Excite’s spider will cycle through recently submitted sites, or existing sites in it’s database. It prefers to start with the home page, and follow links from there. Once it finds 25 pages, it adds them to its index, but then drops all the rest of the pages for that Web site!
Which pages it chooses to keep is somewhat random. Often your important doorway pages will get dropped, and your unimportant pages stay. According to Ray Comstock of http://www.heavymusic.com one solution is to add the following tag to your less important pages that don’t need to be indexed:
<meta name="robots" content="noindex">
This tag is placed in the HEAD section at the top of the page with other meta tags. It tells all search engines that honor the tag (including Excite) to not index this page.
Therefore, if you tell it to exclude all pages except your most important 24 doorway pages, and your home page, then those pages should "stay" indexed. That’s because all other pages can’t be re-indexed because of the "noindex" tag. The best thing to do is to add either visible or hidden links to your doorway pages directory from your home page to insure Excite’s spider can find them easily. Excite prefers you submit only your home page.
For an example of creating hidden links, read this article.
The disadvantage to the "noindex" technique is that it limits you to just 25 pages on many other engines that read the noindex tag. Most engines normally support more than just 25 pages. Ideally, you should create a separate Web site domain for Excite using the "noindex" tags on all but 25 pages, and another domain which allows engine’s to spider the whole site. You could experiment with creating a separate sub-directory of pages for Excite on your existing domain, but Excite generally doesn’t like to rank pages in sub-directories as well as those in the root directory.
WebPosition will help you keep track of what pages are currently indexed, and what ranking each page holds:
http://www.webposition.com
← What is this?
