Nearly all search engines use spiders (also known as robots, their original name) to go out and scour the Web looking for Web pages. These search engine spiders then bring the data back to be indexed by the engine.
Since roughly 1996, individual meta commands have been used on individual Web pages to modify how search engine spiders behave. The most useful of these commands are fairly universal and respected by almost all search engines.
Here are some of the more popular ones and reasons you might want to use them (or not).
<meta name="robots" content="index">
This meta command is one of the most common ones usedâ€”and it is also the least necessary. It tells search engine spiders to come on in and put the page in their index. However, all search engines do this by default anyway.
Basically, if you want to put it in there for fun, be my guest; but this command is not giving you any special treatment. All search engines will index your page, unless you specifically tell them otherwise.
<meta name="robots" content="follow">
The follow command is different from the index command. It basically requests that the search engine spiders follow the links that are on a particular page. Again, however, this piece of code is completely unnecessary, because all search engines are going to follow the links on a page, unless otherwise directed.