The guide will not only help you get the most out of WordTracker, but provides insights into how experts leverage keywords and WordTracker to keep their clients’ sites at the top of the search engine results pages.

WordTracker created a fictional company, Virginia Veg, and asked experts to provide real answers to the problems facing its CEO, Susan Webster. The result is an e-book packed with insight, tips, and techniques on keyword research that you can apply easily to your own website.

Contributors include some of the most recognized names in search marketing and web site optimization.

Creating a robots.txt file:

You need not agonize over how to create a robots.txt file as they extremely simple to make and implement.

First off, a robots.txt file is really just a simple text file that can be created via the standard notepad.exe text editor in Windows, or TextEdit in plain text mode on a Mac. You can even create a robots.txt file under a unix command line. In any of these cases you will want to make sure that the file is saved as (all lowercase) robots.txt and that it is saved under a normal text mode. Using a more complex program will result in a file that also includes formatting information such as font type, font size, etc. which is not needed or read by a search engine.

The three most common items you will find in a robots.txt file are:

• allow
• disallow
• and the wildcard or asterisk: “*”

Normally you would use the “disallow” command so that an engine not index certain areas of your site, while the “allow” command is actually redundant since they will usually follow any other link that you have not prohibited. Finally the wildcard indicates all engines thus if you had a file folder called “images” under the main directory such as: “www.site.com/images/” you might use the following coding if you wished to disallow all spiders from that folder:

User-agent: *
Disallow: /images/

If you wanted to disallow a robot from a particular set of folders, you would use a robot’s name rather than a *. You can even specify individual files. For example:

User-agent: MSNBot
Disallow: /gopher/solutions/

User-agent: Googlebot
Disallow: /beta/private/new_widget_ideas.asp

Other methods of controlling robot access:

In some situations, you might not have the ability to create a robots.txt. There are many affiliate programs where you pay for an ‘in the box’ solution where, in reality, your company may simply be a folder off of someone else’s domain. An example of this would be “www.somecompany.com/your_businessname/” where your ’site’ is the folder “/your_businessname/”rather than a stand alone domain like “www.your_businessname.com”. Since search engine robots normally use only the robots.txt file found under the domain rather than from a folder such as “www.somecompany.com/your_businessname/robots.txt” you may need to use a bit of HTML to accomplish the same goal. However the code will only be used to control how the engine treats the single page in question.

In the meta tags area of your HTML, you can add coding such as:

<META name=”ROBOTS” content=”NOINDEX, NOFOLLOW”>
This tells the robots not to index the page or follow links from that page. However if the robots finds other pages that are linked from other areas of your site, a submission, a link from another site to that page, etc. the pages that do not include the meta tags may still be indexed.

<META name=”ROBOTS” content=”NOINDEX”>
This tells all robots to not index the page.

<META name=”ROBOTS” content=”NOFOLLOW”>
You can also tell them to not follow any links on the page:

<META name=”ROBOTS” content=”NOINDEX, FOLLOW”>
This tag indicates that you might want the robot to follow links on a site map but not index the site map web page itself.

<META name=”ROBOTS” content=”INDEX, NOFOLLOW”>
In unique situations, you might want to link to a site but you might not want the search engine to see your site as ‘promoting’ the site you are linking to.

HTML

Since the meta tag examples above tend to control the whole page, you can instead opt to leave out the meta tag control from your HTML and micro-manage all links or individual links on a page using relevance tags.

This is a normal link to someothersite.com where your site can be counted as “voting” or approving of the some other site page in order to improve the rank of the destination page for the term “blue widgets”:

<a href=”www.someothersite.com/blue.htm”>Blue widgets sold here.</a>

In this example, you might link to the other site, but you do not want a search engine robot to count the link for the sake of rankings:

<a href=”http://www.yetanothersite.com/yelllow.htm” rel=”nofollow”>The truth about yellow widgets.</a>

This is especially useful if you run a forum or have an internet diary or blog. While many of these types of sites welcome comments and allow for links to relevant sites/visitors, there has been a huge number of comment spammers as of late who do not contribute in a positive or constructive way. They often leave a message about their product/service/site and a URL link rather than adding the site or pages conversation. So this can be a powerful tool to block the spammers and in some cases you can remove it for accounts that register with your site, those that you trust, or disallow all posters from receiving link rankings from your site.

Can these techniques be used to “game” the engines?

In the past, it was common place for site owners to create sets of documents that were nearly identical in nature but geared toward one engine or another since search engines have different ideas about what info a ‘relevant’ page should contain. Thus they might block one engine from seeing duplicated content that was geared toward another engine. This was a smart way of doing things in the past unfortunately while search engines attempt to play by the rules, all of these control methods are completely voluntary. While most engines will honor your request to not index the pages or files you do not want indexed, some may still view the content and store it for their own reasons. For example they may choose to compare pages on your site to look for duplicate content and thus cut down potentially spammy content or a whole site altogether.

In Summary

Use robots.txt, the robots meta tag, and rel=”nofollow” with a bit of care and you can control how robots interface with your site quite a bit. There were ways in the past to potentially trick the engines however this is becoming less and less worth while as the search engines evolve. While some sites have been able to get away with these action (since no engine is perfect, yet), we recommend against taking these types of actions due the the potential risks outweighing the potential gains. Instead, we say, play by the rules. Pick and choose who you promote to, which pages you promote to whom, and create themed/grouped areas of your site in order to attain better rankings. If you need to cross link to different areas of your own site which have vastly different topics i.e. if you sold both “maple solid acoustic guitars” and “green silk plants” you might use the nofollow tag. This is a better way to provide good navigation to your customers with out confusing the engines about what topics/keywords/phrases that different areas of your site might be trying to promote and rank on. And if you really, truly do not want an engine to see, index, visit, etc. certain areas or file on your site, password protect them or keep them off line.

For the most part, our previously disabled keywords were deleted while keywords placed on hold were moved into the inactive column. These inactive keywords are now tagged with a short message similar to the following:

…in which case that particular keyword required a bid increase from $0.50 to $1.00 in order to be reactivated (we declined). At any rate, most of our inactive keywords required only a few cents bid-increase to be reactivated. In cases where the ROI justified the slightly added expense, we proceeded to reactivate our on hold keywords.

It appears that being able to easily reactivate on hold keywords by spending just a few cents more per click (as we mentioned, there were exceptions) is possibly the most valuable aspect of Google’s new keyword management system.

Take note that Google states within the reactivation link, increase quality or… suggesting that you can also reactivate keywords by “improving quality.” However, exactly how they are judging quality is still a bit vague since, as you may recall from last month’s report, Google’s new ad-ranking formula is less straight-forward than their original one.

The old formula to determine ad positioning was your Click Through Rate (CTR) multiplied by your Cost Per Click (CPC):

The new ranking formula is far less objective:

Clearly, the new formula is comprised of elements that are unknowable to anyone outside of Google’s inner circle. Regardless, click-through rate (CTR) is still a vital component of the formula so, obviously, you should continue to write compelling ads that entice clicks. Doing so remains the single most important element within your control to lower your overall cost per click.

One major drawback could be that obscure terms may become much more expensive to run—not in every case but certainly in some. Previously, one could bid on obscure and overlooked terms that attracted small amounts of traffic for an extremely cheap cost per click. Finding enough such keywords could generate a nice little traffic stream. Now, however, these keywords could become much pricier, due to the small amount of traffic they typically generate causing them to have poor historical keyword measurements.

The flip side of that rational is that these obscure or ignored terms generated so few clicks that they were often disabled under the old system anyway. So one could reasonably say the change probably won’t actually have much effect on your ad campaigns.

Suspicious people might argue that the subjectiveness of terms like relevancy factor could tempt Google to manipulate algorithms in a self-serving fashion. However, thus far we’ve found the changes have actually made our own campaigns slightly easier to manage with very little effect on what we’re paying to run our AdWords ads.

On the whole, AdWords remains to be the relevancy-based, CTR-driven advertising program it’s always been. Only now there’s less confusion about how to get your perpetually on hold and disabled keywords out of limbo.

Related link:
The Google AdWords Performance Monitor: How to Maintain Quality, Cost-Effective Keywords

The above article is reprinted with permission from Planet Ocean’s SearchEngineNews.com, copyright 2005, and distributed with permission by WebTrends maker of WebPosition, the award-winning software that helps track and improve your search engine rankings. You may download a FREE trial copy of WebPosition from: http://www.webposition.com/trial/