Latest Articles

Seed Sites are One of SEO’s Best Kept Secrets

August 20, 2010 By Sue Spiker

Part of a search engine’s secret formula is what domains it chooses to use as seed sites, the entry points through which it deploys its web crawlers to explore and index the Internet.  Knowing exactly what domains it chooses as seed sites could tell us exactly where to make sure our websites are listed and where to focus our linking strategies.

Helpful information for those of us on an eternal quest for a higher page rank.

No such comprehensive list is publicly available, for two reasons.  First, search providers use a multitude of seed sites, and they’re not committed to using the same sites each time.  Domains such as Yahoo Directory and DMOZ are likely candidates, but using a variety of domains increases the chances of discovering new URLs and finding out which URLs have expired, which helps keep its search index up-to-date.  Therefore, that list would constantly be in flux and not entirely accurate at any given date.

Second, search providers are historically (and understandably) protective of their trade secrets and are not expected to volunteer such key information any time soon.  Basically, they could tell us, but then they’d have to…well…you know.

And we can’t really blame them.  Imagine what would happen if such information fell into unscrupulous hands.  Spammers and hackers the world over would start gaming the system before we could say “Let me look that up online.”

So give your website the best exposure you can by submitting it to as many appropriate directories and niche portals as possible.  While you’re at it, use social media to get your site out there.  Search engines will find your website if you just keep spreading the word by practicing solid, white hat SEO techniques.  And when they do, make sure that you’ve done a thorough job of optimizing your content, coding and structure, including creating a sitemap.xml file to tell their web crawlers exactly where to go next.

Teaching New Tricks to Old Robots, New Robots.txt Command for SEOs.

May 8, 2007 By Webposition SEO Team

By Scott Goodyear

Sitemaps.org, a collaborative effort by Yahoo, Microsoft, and Google, has announced a change in robots.txt usage. The new change should allow you to tell participating saerch engines where you keep the sitemap on you site. The usage is fairly simple and it is likely that most other engines will soon support this standard.

How To Use The Robots.txt Sitemap Tag

Normally you will want to use a robots.txt file to tell search engines to stay out of certain files or folders. Here is an example where Google’s own robots.txt file is excluding many folders from being indexed by their own robot, Googlebot, as well by other search engine robots by utilizing the user agent “*”:

User-agent: * Allow: /searchhistory/ Disallow: /news?output=xhtml& Allow: /news?output=xhtml Disallow: /search Disallow: /groups Disallow: /images Disallow: /catalogs Disallow: /catalogues Disallow: /news Disallow: /nwshp

But now you can also tell a search engine exactly where you keep your sitemap by including a bit of code in the robots.txt file. For example there is a “sitemap” link at the bottom of most of our MarketPosition pages but we could also tell search engine robots about our HTML sitemap or tell them about the text version of our site map by including the new sitemap command in our robots.txt file (you don’t have to have both an HTML and text version, we’re just testing to see which seems to work better) like this:

User-agent: * Allow: Sitemap: http://www.marketposition.com/textsitemap.txt

In the past, the robots.txt file was pretty one dimensional in purpose. It served only to exclude robots from certain areas of a page or site. If you wanted to exclude them from indexing your images directory, you used a disallow command. If you wanted them to slow down because they were hammering your site with page requests as they crawled, thus slowing your site down, some engines allow for a delay option in the robots.txt file. Below, I’ve added a command to slow Microsoft down when indexing:

User-agent: * Allow: User-agent: msnbot Crawl-delay: 60 Sitemap: http://www.marketposition.com/textsitemap.txt

Further Reading And Discussion

If you want to read more about robots.txt and site maps, consider the following resources:
Controlling Search Engine Robots With robots.txt and Other Methods.
How to Identify Legitimate Search Robots.
The Robotstxt.org and Sitemaps.org web sites.
For more discussion on these topics, consider:
HighRankings.com and WebmasterWorld.

How to Identify Legitimate Search Robots

December 7, 2006 By Webposition SEO Team

by Richard Drawhorn

Everybody knows that search engines use search robots to locate and index content, and webmasters should certainly allow them to do so. However, given the fact that not all robots are harmless, how do you validate that a robot is authentic? This topic has come up on the search blogs for both Google and MSN recently, and in this brief post I’ll summarize their advice on how to identify their robots.

When viewing your server logs, you’ll find entries for each visit to your web site and the corresponding IP address. If the visit was a search robot, it will have a user-agent entry like Googlebot or MSNBot for example. Each search engine has its own user-agent, but these are not sufficient to identify the robot because any spammer can name their robot Googlebot if they so choose.

I took at look at the documentation posted by Google, Bing and Ask.com, and they all agree that the best way to identify a search robot is as follows:

1. First do a Reverse DNS Lookup to confirm the hostname associated with the robot’s IP address. There are several free tools (like this one) on the web that can be used to do the reverse DNS lookups, but you’ll probably find it easier to use command line tools to gather this information.
2. Verify that the hostname is correct:
   • For Google, it should be in the googlebot.com domain (such as crawl-66-249-66-1.googlebot.com for example).
   • For MSN, it should be in the search.live.com domain (such as livebot-207-46-98-149.search.live.com for example).
   • For Yahoo!, it should be in the inktomisearch.com domain (such as ab1164.inktomisearch.com for example).
3. Finally, do a Forward DNS Lookup on the hostname to confirm that it matches the IP address. This last step will verify that the name itself is accurate.

If you do find a robot that has been been disguising itself as a legitimate search engine crawler, you’ll probably want to block that robot’s access to your web site (which can be easily achieved by configuring your web server appropriately).

Conclusion
Verifying that robots visiting your web site are authentic search robots is in any webmaster’s best interest. Many illegitmate robots do not follow the conventions defined by the robots.txt file and can cause performance issues for a web site if allowed to spider the site unimpeded.

Should I Submit Again?

April 24, 2006 By Webposition SEO Team

By Scott Goodyear

Have you launched a new web site for yourself or a client, submitted to the search engines, waited a few months, and were left wondering if the site had gotten into the search engines? Maybe you’ve found the URL you’ve submitted but don’t know if they’ve only picked up the URL, or if they had picked up the content from your pages? You may also be wondering from time to time whether you should resubmit or not? In this article, we will explore some of the methods that you might use to answer these questions.

Extended The Invitation To Visit.

When working on the content of a site, gaining inbound links, searching for link partners, etc., you may have submitted a few pages to the major search engines and received a vague notice that your site has been added to a list of sites to potentially crawl and index. Here are a few examples of messages you may have seen:

A few months will likely pass and at that point, it is usual that the average site’s submitted pages will not be listed in the top100 let alone top 1000 rankings. Should you resubmit? The answer to this question may lay in answering this: was the site’s home page, or at least a few pages within the site even indexed?

Try searching with a few URLs as ‘keywords’. For example: www.marketposition.com/blog/archives/2004/12/index.html on Netscape.com This can be an important step since you may not find the URL under a specific keyword or phrase, but you will want to at least know if the engine is aware of the site. This can be a bit time consuming since you might need to check for URLs with and without the “www.” in them. You may wish to consider using some of the advanced search operators like:

site:www.site.com
inurl:www.site.com
domain:www.site.com

The actual search operators can vary based on the engine you are using, but most sites will offer an advanced search option or domain search option if you do not wish to bother with search operators. There are also engines like Yahoo! that have an exploration tool to simplify this process.

Did An Engine Just Knock At The Door?

If you have a web analytics package or know how to review your site’s log files, you can see whether a search engine robot has been visiting your site. Many search engines robots have obvious names like Google Bot, MSNbot, etc. or unique names like Slurp, rather than the simple IP address/browser name combinations that most of your human visitors will have. If you are taking part in a search engine’s paid advertising program, this can mean that your site will be indexed not only by a normal search spider but could also be analyzed by an advertising program spider. While advertising and submission services will not typically cause your site to gain an extra rankings boost, it can mean that your site will be indexed faster than letting the spider naturally crawl your site and incidentally, may mean that your site information is updated more often than a non-advertiser.

I Can’t Believe They Still Have THAT In The Index!

Some time has passed and you’ve likely had a few pages that have been indexed. From time to time, engines will return to your site to check for updates automatically. When others link to your site, the engines can also follow those links back to your site and re index your site as well. This is why larger, more popular sites like CNN.com, BBC.co.uk, etc. are often quite up to date in the engines. Yet when looking at listings from your own site, you may find information that is out of date or different from the content that is currently on the page.

If your indexed content is out of date, first check your analytics package/log files as mentioned previously. Has the engine been by in the past few months? If so, it may simply take a bit longer for the engine to update its database and spread the updates throughout its various data centers. If you do not have this type of option available to you, some engines will allow you to check their ‘cache’ directly from their search. Here is a search in MSN for the URL:
http://www.marketposition.com/blog/archives/2005/09/controlling_sea.html

Search, then click on the ‘Cached page’ option.

On the next page, at the top there is a note about when MSN had last examined this page. Since they have visited recently, I would not resubmit this even if the contents of the page had changed on say 4/7/06. The engine’s robot should come back to visit the page in the future whether I resubmit or not. If the date is quite old, such as 3-6+ months out of date, you may wish to resubmit if the page has updated information. If the content on the page is the same as the cached copy, there is no reason to resubmit and resubmission at that point will not help to improve your rankings.

How ‘engine friendly’ is your site? Some sites that rely heavily on flash, javascript, or other technologies are harder to interpret and index than those based on plain HTML/text. Search engines can choose to display listings from a directory like The Open Directory Project (DMOZ.org), if a search engine has a problem reading your page. You can try to get your DMOZ listing updated however it might be better to fix your page so that it is more search friendly. The text based Lynx web browser is ideal for displaying a very basic text version of your page, similar in many ways to what an engine might see. If you do not have a DMOZ listing you may simply end up with an entry for your URL, but no description. This is sometimes referred to as a ‘partial listing‘.

Many engines have multiple data centers. While the engines often update their data centers on a continuous basis, from time to time you could notice that a page seems to be partially indexed, such as a URL with no text description. If you perform a unique text string search, you may find that the page’s content has indeed been indexed, but perhaps the information has not yet been linked to your normal listing due to a canonical issue (i.e. your site was indexed twice, once under www.site.com and once under site.com where one or the other shows up with the text listing). If you do find that one or the other is listed with text and the other is not, you may need to work on creating a 301 redirect rule for your site.

So how do you perform a URL verification search? Again, I will use this page: http://www.marketposition.com/blog/archives/2005/09/controlling_sea.html
I can look for unique text that other sites would not typically have. I could probably use the title of the article, a combination of the author name and a keyword from the page, or other text information that is fairly unique like:

‘Controlling Search Engine Robots With robots.txt and Other’ Google Yahoo MSN
‘Scott Goodyear Robots.txt’ Google Yahoo MSN
‘conserve bandwidth (data transfer) since some engines will completely skip files and folders indicated’ Google Yahoo MSN
(Some words like and, or, the, .com, and others are known as noise words or stop words which most engines will ignore because they are so common.)

As an aside, if you have a list of several URLs that you use to periodically verify unique text, you can check these URLs from time to time (once you have first verified that the URL is pulled up with your unique text) and submit to the engines if your URL no longer shows up with the unique text. At that point you will know that it is a good time to resubmit. If you are using Web Position 3 or the newest Web Position 4 software, a URL verification feature is built into the Reporter and Submitter modules. Simply create your Reporter mission as normal, follow the URL verification steps from just above, and if the URL is found, enter this information into the Reporter mission. Save and close this mission. Create a Submitter mission and check the URL verification option. Save and close this mission. Start a new Scheduler event, set it to run say once a month with these two new missions and WebPosition will run once a month to make sure that your pages are still indexed and if not, resubmit them for you.

How to Leverage Expired Domains to Increase Your Rankings and Traffic

May 15, 2004 By Webposition SEO Team

Written by Matt Paolini / Edited by Brent Winters

New sites appear on the Web each day at a blistering pace. Just as thousands of new sites are born each week, according to DomainsBot.com, one hundred thousand other domains will disappear. What happens to these domain names? Unless the owner continues to pay rent on the defunct domain, the name will eventually expire and be offered back to the public.

Why Can’t I Get Indexed by the Search Engines?

March 15, 2001 By Webposition SEO Team

Unfortunately, this is an all too common question. If it makes you feel any better, you’re not the only one frustrated about the length of time it takes to be indexed, or the many pitfalls involved. It often takes anywhere from two days to as much as six months to be listed on a search engine. For example, last month Excite finally updated its index for the first time since last August! Luckily, Excite is the most extreme case lately, but waiting several weeks to a month can also be extremely frustrating.

The WebPosition Submitter report will give you current time estimates for each engine so you’ll know what to expect. However, an engine at any time could choose to delay their indexing beyond the “norm” for maintenance or other reasons. On the flip side, you could get lucky and submit just a couple days before an engine does a complete refresh of their database. Therefore, submission times can never be an exact science since we’re all ultimately at the mercy of the engine.

If you’ve submitted your site and have waited the estimated time to be indexed and there’s still no listing, what do you do now?

Here are 16 tips that should help you solve this problem:

1. First, be sure you’re not already indexed but just don’t know it. Unfortunately, none of the major engines are kind enough to e-mail or notify you as to if and when you’ve been indexed. The method to determine if a page or domain has been indexed varies from one engine to another, and in many cases, it’s difficult to tell for sure. Never assume that you’re not indexed just because you searched for a bunch of keywords and you never came up in the first few pages of results. You could be in there but buried near the bottom. In addition, it’s not very practical to check the status of a number of pages on each major engine each week. Fortunately, WebPosition has a URL verification feature in the Reporter that makes this process much easier. Each time you run a mission, it will report which URLs exist and do not exist in each engine.
2. Make sure you have uploaded the pages to your site before submitting them. This one seems obvious, but submitting a page that does not exist or submitting with a subtle typo in the URL is a goof we might all make at one time or another. If you’re using WebPosition’s Submitter, there’s a checkbox on tab 2 that forces WebPosition to verify that all your URLs are valid before submitting them.
3. If you have information inside frames, that can cause problems with submissions. It’s best if you can create non-framed versions of your pages. You should then submit the non-frames versions of your pages which can of course point to your framed Web site. Alternatively, you can enter your relevant text within the NOFRAMES area of a framed page which most search engine spiders will read.
4. Search engine spiders cannot index sites that require any kind of registration or password. A spider cannot fill out a form of any kind. The same rule applies regarding indexing of content from a searchable database, because the spider cannot fill out a form to query that database. The solution is to create static pages that the engines will be able to find.
5. Dynamic pages often block spiders. In fact, any URL containing special symbols like a question mark (?) or an ampersand (&) will be ignored by many engines.
6. Most engines cannot index text that is embedded in graphics. Text that appears in multimedia files (audio and video) cannot be indexed by most engines. Information that is generated by Java applets or in XML coding cannot be indexed by most engines.
7. If your site has a slow connection or the pages are very complex and take a long time to load, it might time out before the spider can index all the text. For the benefit of your visitors and the search engines, limit your page size to less than 60K. In fact, most Webmasters recommend that your page size plus the size of all your graphics should not exceed 50K-70K. If it does, many people on dial up connections will leave before the page fully loads.
8. If you submit just your home page, don’t expect a search engine to travel more than one or two links away from the home page or the page that you submitted. Over time they may venture deeper into your site, but don’t count on it. You’ll often need to submit pages individually that appear further down into your site or have no link from the home page.
9. If your Web site fails to respond when the search engine spider pays a visit, you will not be indexed. Even worse, if you are indexed and they pay a visit when your site is down, you’ll often be removed from their database! Therefore, it pays to have a reliable hosting service that is up 99.5% of the time. However, at some point a spider is going to hit that other 0.5% and end up yanking your pages by mistake. Therefore, it pays to keep a close eye on your listings.
10. If you have ever used any questionable techniques that might be considered an overt attempt at spamming (i.e., excessive repetition of keywords, same color text as background, or other things that the WebPosition Page Critic warns you about), an engine may ignore or reject your submissions. If you’re having trouble getting indexed in the expected amount of time, make sure your site is spam-free.
11. If your site contains redirects or meta refresh tags these things can sometimes cause the engines to have trouble indexing your site. Generally they will index the page that it is redirecting TO, but if it thinks you are trying to “trick” the engine by using “cloaking” or IP redirection technology, there’s a chance that it may not index the site at all.
12. If you’re submitting to a directory site like Yahoo, Open Directory, NBCi, Looksmart, or others, then a human being will review your site. They must decide the site is of sufficient “quality” before they will list it. I recommend you read the submission guide on the directory tab of the WebPosition Submitter. It contains tips to improve your chances of obtaining a good listing on these directories.
13. A number of engines no longer index pages residing on many common free web hosting services. The common complaint from the engines is that they get too many “junk” or low-quality submissions from free web site domains. Therefore, they often choose not to index anyone from those domains or they limit submissions from them. It’s always best to buy your own domain name (very important) and place it on a respected, paid hosting service to avoid being discriminated against.
14. Some engines have been known to drop pages that cannot be traveled to from the home page. HotBot has been rumored to do this. You may want to consider submitting your home page that links either directly or indirectly to your doorway pages.
15. Make sure you’re submitting within the recommended limits. Some engines do not like more than a certain number of submissions per day for the same domain. If you exceed the limit, you may find that all your submissions are ignored. Fortunately, WebPosition’s submitter will warn you regarding current limits and recommend you stay within them. Some submission consultants feel it is dangerous to submit more than ONE page a day to a engine for a given Web site. For those who wish to be ultra-conservative in their approach, the WebPosition Submitter includes a checkbox to limit submissions to one URL per day per engine.
16. Last but not least, sometimes the engines just lose submissions at random through technical errors and bugs. Therefore, some people like to resubmit once or twice a month for good merit in case they do lose a submission. Certainly if you’ve followed all the “rules” and are still not listed, re-submit! Sometimes a little persistence is all that’s needed.

If any of the above scenarios apply to your submission, you should make the necessary adjustments and re-submit. If that still does not work, you should consider e-mailing or calling the search engine and asking them politely why you have not been indexed yet. Sometimes they will reply back with “Sorry, there was a problem with our system and I’ve now made sure you’ll be indexed within the next couple days.” Or, sometimes they’ll tell you why you were not indexed. In other cases, they will ignore your e-mail and you’ll have to keep e-mailing or calling them until they respond. Still, it’s definitely worth the effort to get your site listed with the major engines assuming you also take the time to optimize your pages so you’ll achieve top rankings.

Is Submitting Manually Better?

By Webposition SEO Team

If you have a lot of extra time on your hands, you could submit your pages manually to the search engines each time you change your content. However, this can be time consuming if you are trying to optimize for many search engines on a regular basis. It is also prone to error since you must re-type the URLs each time correctly, or at least paste them in one by one.

Some people are wary of automated submission tools because they’ve heard they are also prone to errors. In the case of some submission products or services on the market, this is a valid concern. One problem is that many submission tools fail to look for a “success” message after the submission is made. They just assume that all will be “cool” and they move on to the next submission. WebPosition Gold, however, will not report a submission as successful unless it receives back a specific string from the search engine indicating the submission was accepted.

A few people shy away from automated submissions for fear that the submissions will be sent to the search engine so quickly that it will “red-flag” them as using an automated tool. However, with WebPosition, you may choose to submit one page per day per engine if you like so that the speed of the submission becomes a non-issue. In addition, WebPosition emulates a common browser so that your submissions look no different to the search engine than if you’d used Internet Explorer to submit.

How Often Should I Submit?

By Webposition SEO Team

As a rule you will only want to resubmit your Web site each time you update or change it and only if you are not ranking well. The search engines will not be instantly aware of the changes you made to your Web site. Therefore, you will generally want to tell the engine about your changed pages by re-submitting them. You can go to each search engine and individually submit each of your pages, or you can use WebPosition Gold’s Submitter to create Missions to do it automatically.

Unfortunately, there are pitfalls to submitting since each engine has their own set of rules and submission limits. Unlike many other submission products and services, WebPosition will try to warn you about known limits so you can avoid being penalized or having your submission ignored. If you try to submit too many pages at once, WebPosition will prompt you and suggest submitting only some of the URLs for that day. Surprisingly, the engines penalize for exceeding these limits but rarely ever warn the Webmaster if they exceed them.

Also, resubmitting before the engine has had time to index your page is not necessary. After submitting, it will take anywhere from a couple of days to months for your submission to be integrated into their database (i.e., indexed), and for your ranking to change. The amount of time varies for each search engine. The WebPosition submission report generated after you submit will give you estimated times for each engine so you know what to expect.

Regarding directories such as Yahoo, you will never need to resubmit unless the content of your Web site changes significantly or the description they chose for your listing is not accurate. The method to update or change your listing on a directory will vary. See the Directory Submission Guide of WebPosition’s Submitter for details.