« A Review of Web Site Accessibility Guidelines | An Introduction to the Social Scene: Social Bookmarking and Tagging »

July 28, 2006

Yahoo! Introduces New Search Crawler

by Richard Drawhorn

In a recent entry in their search blog, Yahoo! announced the introduction of a new version of their search crawler, Yahoo! Slurp. The new spider is "faster" and "more efficient" according to Yahoo!, which is good news for web site owners since it should consume less bandwidth while crawling their site.

Not much information is given in the blog post, but the following statement is somewhat of a red flag: "you may see some shuffling of the pages that are included in the index and some changes in ranking as well." Web site owners might want to keep a close eye on their rankings on Yahoo! over the next few weeks to see if they are affected by any changes resulting from this change. A convenient way of tracking rankings is to use a tool such as WebPosition's Reporter. If you find that specific pages have dropped out of the index, then it would be a good idea to analyze the page to ensure it's well optimized.

This might also be a good time to review some of the basic information about the new Yahoo! spider by reading the FAQs on their web site. You'll find useful information there including how the spider handles redirects, or the robots.txt file for example. For more information about controlling the behavior of search engine spiders with the robot.txt file, read this article.

Digg.com    del.icio.us    furl.net    newsvine.com    reddit.com    Yahoo! Myweb   ← What is this?

Read more articles in the Yahoo! topic category.

« Previous | Next »