The Robots Meta Tag is used mainly when you don’t want search engine spiders to index a webpage or to follow the links on a webpage. Another way to control this is through a robots.txt file. But here I’ll just discuss the Robots Meta Tag.

When a webpage is indexed in search engines it means that the page is in the Search Engines database. So when someone searches for keywords related to your indexed page, your page will show up in the Search Engine Results page. The follow directive tells the Search Engine to follow the links on that page as well.

The default behavior if this tag is not specified is for the search engines to index the webpage and to follow the links as well. So it’s not necessary and not recommended to add the below code to a page if you want it indexed and followed.

<meta name=”robots” content=”index,follow”>

If you want a webpage indexed but don’t want the links on that page to be followed then you’d use the directive below. Generally if you have an outgoing link but don’t trust the link, then it’s wise to use the nofollow. To specify the nofollow for an entire page, use the Robots Meta Tag below. This would apply to all links on the page.

<meta name=”robots” content=”index,nofollow”>

If you want specific links not to be followed, not all links on the page, then you wouldn’t use the above directive. Instead You can specify nofollow per link using the below link link tag example.

<a href=”contact-us.html” rel=”nofollow”>Contact Us</a>

Most websites contain pages that we want the search engines to index so that internet surfers may find them when looking for information on our website. But there are times when you don’t want a page indexed.

For example you may create a website just to share family photos not with everyone on the internet but just with family and friends. Another example is when you create websites for your customers that you share only with customers and not with the public. In these cases it’s a good idea to add the Meta Robots Tags directive for noindex and nofollow like the example below.

<meta name=”robots” content=”noindex,nofollow”>

Then there are cases where you don’t want a page to be indexed but you want the links to be followed. Good pages to do this on are utilities pages, like contact us, privacy policy, and terms of use pages. Use the below directive in these cases.

<meta name=”robots” content=”noindex,follow”>