We released WebPosition back in 1997 as the first software tool to help businesses track and improve their search engine rankings. WebPosition broke new ground at the time and continues to do so today as we adapt the product to a continually evolving marketplace. Since WebPosition's introduction and the founding of the MarketPosition newsletter six years ago, I've witnessed countless industry changes. I've also witnessed dozens of misguided predictions in the search engine industry.

For example, I remember in the early days of the industry when Yahoo! was more popular than all the other search engines combined. Many were concerned that Yahoo! would soon crush all other competitors with their immense popularity, their giant catalog, and their early lead. I felt such predictions were overblown, since good ol' competition and advancing technology have a way of evening things out over the long run.

Later, Inktomi started an innovative practice of licensing its results to one search engine portal after another. For a time, people were predicting that nearly all search traffic would be served by the Inktomi juggernaut. Why optimize for other engines when Inktomi would soon control it all? Of course Inktomi eventually lost steam and those worries proved to be unfounded.

I also witnessed the rising power of directory-based engines such as LookSmart, Open Directory, Yahoo!, and the now defunct Snap. For a time, search engine optimizers were worried that optimizing their page content would become futile, as "all search results were soon going to be dominated by the human-reviewed catalogs." What need did we have for crawlers when humans could do so much better at cataloging the information on the Web?

At the time, I received a number of e-mails from readers predicting the doom of WebPosition and crawler-based search because of the growing popularity of these catalogs. However, I pointed out that WebPosition had evolved to provide valuable advice for obtaining top rankings in these catalogs and in providing effective tracking. In addition, directories were only one part of the puzzle. While they did a good job in organizing similar types of content, they failed to provide the best results for the majority of user queries. Therefore, while directories would play an important role, they would never dominate the search engine results.

In the midst of these changes, Goto.com arrived on the scene, only to attract relatively little attention at first. After all, a search engine called Open Text had previously tried to sell search engine listings and had failed after a bungled PR campaign and intense criticism. As we now know, Open Text was simply ahead of its time in its attempt to add a commercial element to search.

Goto.com, now renamed Overture, realized that carrying E-Bay's successful bidding model over to the area of online advertising made perfect sense. Bidding on search listings allowed the market to establish the true price of a targeted ad in real-time and at a lower transaction cost. This was when banner ads were declining in their effectiveness to reach the consumer, yet remained grossly overpriced for most advertisers. That change opened a door of opportunity for Overture.

Today, I still hear someone from time to time citing the demise of free crawler-based search as well as the feared end to WebPosition. They point to Overture's success as evidence of how all search results will soon be dominated by paid listings. "Why bother with the free crawler-based results?" they say ask, as the Web is now going to be completely commercialized. However, these people fail to recognize that it is ultimately the consumer controlling the market, not the advertiser. While advertisers may like the convenience and economy of bidding for their rankings, this alone will not guarantee the model's success or make it dominate the space.

While consumers find paid results acceptable for finding certain kinds of information, it fails to provide the best and most comprehensive results for many queries. Consumers continue to perceive sponsored results as commercial advertisements, clicking on the main search results a greater percentage of the time. This happens even though the sponsored results appear first. It's not only the perception that these results are basically commercial advertisements, but also the fact that they are not as targeted as crawler-based results. Crawler-based results try to serve up the best matches to a query regardless of how much money the vendor may be willing to spend.

Google recognized the continuing need for "pure" search when it made its late entry into the search engine space in 1998. It then rose rapidly against players hundreds of times larger than itself by giving consumers what they really wanted: Faster, more accurate search results. Google did this by first emphasizing its free, crawler-based engine and then later supplementing those results with directory-based links. They also added sponsored ads around the edges of the primary content.

With that said, what will happen now in the industry? Certainly change is a constant. It's important to stay on top of changes in the SEO industry by reading MarketPosition and similar newsletters. You'd also be wise to leverage the more detailed advice offered by WebPosition Gold's Page Critic.

Most people don't realize that simply going from freely available information to a paid solution puts you a cut above the majority. In addition, don't set yourself up for failure by assuming that since your competition may be trying to obtain top rankings too, that you should not even bother. The key is that you need to do it better and smarter than they do. Many times this simply means being a little bit better informed than the average person and being willing to go the proverbial extra mile.

As for the technological outlook for the industry, you will never see the search results dominated solely by paid ads, directories, crawler-based results, or featured listings. While each technology has proven itself to be useful, none provide a total solution to the consumer. I've said this for years and I will say it again today.

We'll continue to see multiple technologies employed in the quest to provide better and more relevant search results to the consumer. Crawler-based results will continue to be perceived by the public as the "core" part of search since they crawl and index the entire content of the page, providing the most comprehensive and least biased results.

This means that your company will want to continue to optimize your site's content so that it ranks near the top while also remembering important off-page criteria such as link building. At the same time, you should consider making paid listings a part of your over-all marketing strategy while being careful to design and purchase only 'optimal' ads.

Paid listings must be carefully designed and measured to turn a profit for your particular business. Doing this haphazardly can cost you money with each and every click-through. You'll also want to educate yourself regarding the strategies and techniques for achieving the best placements in the human reviewed directories.

On all three of the aforementioned fronts, today's WebPosition Gold helps you to optimize and track your total search engine visibility. This makes the tool as valuable today as it was when it was first introduced in 1997 -- if not more so.

In addition, as with most maturing industries that operate in a free market economy, we will continue to see a move toward consolidation. This is to achieve further economies of scale. However, this will continue only up until a point. That's because the government and the public both know that monopolies most often prove bad for the consumer. Therefore, consolidation will ultimately be kept in check through industry regulation and market forces.

Using history as a guide, this means that ultimately three or four major players are destined to control a majority of the general search engine market. The dominant players, at least for the foreseeable future, will be Yahoo!, Google, and MSN. Lycos, LookSmart, AskJeeves, About.com, and other brands will play an important, but auxiliary role.

With the continually growing volume of traffic on the Web, it can still be worthwhile to invest time with the auxiliary engines and brands. However, be careful to avoid falling into the widespread myth that submitting to hundreds or thousands of engines will bring significant traffic to your site. As has been the case since the beginning, most of these engines don't command enough traffic to justify optimization. If you're not prepared to specifically optimize one or more pages for each engine that you submit to, you shouldn't expect to achieve any traffic from that submission.

Since technology and market conditions change so quickly, it's not impossible for an auxiliary search engine player to later unseat one of the major players from their top spot. However, this will admittedly become more infrequent as the major players establish themselves.

Students of history know that the U.S. automotive industry eventually consolidated into three mega-companies: GM, Ford, and Chrysler. However, this did not stop other players such as Toyota, Honda, Nissan, and others from entering the U.S. market late in the game and competing effectively in an industry with higher barriers to entry than today's search market. In addition, other automotive companies sprang up to address niche markets such as high-end sports cars, farm equipment, luxury cars, specialty trucks, and so forth.

In the same manner, I predict that while we will see three to four companies owning the majority of the general search market, auxiliary players will continue to hold their own important roles in the coming years. For example, engines targeting the needs of consumers in specific countries will be one of these areas. A portion of this market will be addressed by specialized versions of the major engines. However, local players tend to know their market better than their global counterparts. Therefore, one would expect some of these to continue to compete in various local and niche markets.

Some engines will survive by being audience and task-specific. For example, shopping engines are growing in popularity because they are designed to quickly find products at the best prices from the best vendors. Other engines that focus on finding certain kinds of specialized information will continue to have a market. Many small businesses can prosper simply by doing well in one of these niche engines. As the audience for these niche markets grows, the related engines will be included in WebPosition.

Another important trend to recognize is that each year millions of new consumers from around the world go online for the first time. This significantly increases the number of queries made on the world's search engines. This, combined with the growing search engine consolidation, means that each top ranking a company achieves will be much more valuable than in past years.

One top ranking can now bring in significantly more traffic than the same top ranking in past years. Whereas a company may have had to achieve thirty top rankings across a wide number of engines to generate $10,000 a month in revenue, that same company may only need to achieve a handful of rankings on a couple of major engines to generate the same kind of sales volume.

This change has turned SEO in recent years into a big business. Whereas in the past many companies had not even heard of SEO, most companies are now at least familiar with the concept. The leaders in each industry are beginning to see the importance of promoting to this growing market and are consequently developing search engine marketing (SEM) strategies.

In the early days of SEO, there was a cap on the total traffic you could expect to gain from the major search engines. The time involved was also significant since there were far more engines to optimize for. Today, the potential payoff is much bigger than it ever has been.

While one must spend extra effort today building external links to a site, fewer hours are spent targeting a previously larger number of search engine crawlers. Therefore, search engine marketers today can afford to invest more of their time on the remaining dominant engines. The smarter ones will target niche keywords and phrases to reduce their competition to a manageable level. Furthermore, niche keywords that once did not generate enough search activity to be worthwhile are now surprisingly successful because of today's larger traffic volume.

Your best strategy, as with any kind of marketing, is to target niches that are under-exploited by your competition. That's how the game is played and what we've preached from the beginning.

For further help in this area, be sure to read Brad Konia's article this month about how to attract the right kind of crowd. In addition, it's more important than ever to take advantage of the best tools and technology to gain top rankings on the search engines. I'm admittedly biased in this regard having been the chief architect of WebPosition. However, there's certainly wisdom in starting with the tool that has the greatest experience and the most proven track record.

Download the WebPosition Free Trial
Visit our site to learn more about WebPosition

                 ← What is this?

---