Nearly all search engines use spiders (also known as robots, their original name) to go out and scour the Web looking for Web pages. These search engine spiders then bring the data back to be indexed by the engine.
Since roughly 1996, individual meta commands have been used on individual Web pages to modify how search engine spiders behave. The most useful of these commands are fairly universal and respected by almost all search engines.
Here are some of the more popular ones and reasons you might want to use them (or not).
<meta name="robots" content="index">
This meta command is one of the most common ones used—and it is also the least necessary. It tells search engine spiders to come on in and put the page in their index. However, all search engines do this by default anyway.
Basically, if you want to put it in there for fun, be my guest; but this command is not giving you any special treatment. All search engines will index your page, unless you specifically tell them otherwise.
<meta name="robots" content="follow">
The follow command is different from the index command. It basically requests that the search engine spiders follow the links that are on a particular page. Again, however, this piece of code is completely unnecessary, because all search engines are going to follow the links on a page, unless otherwise directed.
Take the first step (it's free).
You may also like:
- Younger and Wordier: How Gen Z's Search Behavior Is Different
- The Most Effective Link-Building Strategies According to SEO Experts
- Three Off-Site Tactics That Seriously Boost SEO Traffic
- SEO vs. PPC: Which Search Marketing Strategy Should You Choose? [Infographic]
- Four Ways to Prepare for Voice Commerce