Limited Time Offer: Save 30% on PRO with code WOOHOO »

Real-World Education for Modern Marketers

Join Over 600,000 Marketing Professionals

Start here!
Text:  A A

All Search Engines Love Spiders: How Meta Commands Can Help You Love Them Too

by Scott Buresh  |  
April 15, 2008

Nearly all search engines use spiders (also known as robots, their original name) to go out and scour the Web looking for Web pages. These search engine spiders then bring the data back to be indexed by the engine.

Since roughly 1996, individual meta commands have been used on individual Web pages to modify how search engine spiders behave. The most useful of these commands are fairly universal and respected by almost all search engines.

Here are some of the more popular ones and reasons you might want to use them (or not).

<meta name="robots" content="index">

This meta command is one of the most common ones used—and it is also the least necessary. It tells search engine spiders to come on in and put the page in their index. However, all search engines do this by default anyway.

Basically, if you want to put it in there for fun, be my guest; but this command is not giving you any special treatment. All search engines will index your page, unless you specifically tell them otherwise.

<meta name="robots" content="follow">

The follow command is different from the index command. It basically requests that the search engine spiders follow the links that are on a particular page. Again, however, this piece of code is completely unnecessary, because all search engines are going to follow the links on a page, unless otherwise directed.

Sign up for free to read the full article.Read the Full Article

Membership is required to access the full version of this how-to marketing article ... don't worry though, it's FREE!


We will never sell or rent your email address to anyone. We value your privacy. (We hate spam as much as you do.) See our privacy policy.

Sign in with one of your preferred accounts below:


Scott Buresh is founder and CEO of Medium Blue, an award-winning search engine optimization company.

LinkedIn: Scott Buresh

Rate this  

Overall rating

  • This has a 4 star rating
  • This has a 4 star rating
  • This has a 4 star rating
  • This has a 4 star rating
  • This has a 4 star rating
5 rating(s)

Add a Comment


  • by Sam Mooney Tue Apr 15, 2008 via web

    Good article, thanks. I've been seeing a lot about robot tags lately, now I understand it.

MarketingProfs uses single
sign-on with Facebook, Twitter, Google and others to make subscribing and signing in easier for you. That's it, and nothing more! Rest assured that MarketingProfs: Your data is secure with MarketingProfs SocialSafe!