Speak to an old school developer and some will confidently tell you that no-following or disallowing the page in robots.txt is the best way. Despite the confusion, it is pretty straight forward. Take a look at how to easily remove a single page from the search results below. First, let’s clear up what everything actually does. (If you already know all this, jump to the last section ‘But how do I actually remove a page?’).
Robots.txt
What is Robots.txt? - The robots file allows you to control where the crawlers go on your site. The fancy name for this is the ‘Robots Exclusion Protocol’ and it lets you specify which parts of your site shouldn’t be seen or processed.
While there are directives that allow you to no-index a page in the robots file - there has been mixed messaging from Google about whether they want you to do this.
Verdict: Do not use the robots.txt file to remove a page. It is not an effective way to remove an existing page from the index
Webmaster Tools URL removal tool (Search Console)
What is the URL removal tool? - Google’s search console is a treasure trove of useful tools and insights into your website, everyone who owns a site should have it. The removal tool allows you to temporarily remove a page from the index.
Verdict: Effective in the Short term – however, to keep the page from being re-indexed… keep readin’
Meta Robots Noindex
What is the Meta Robots Noindex meta tag? - This is a HTML tag that you can add to any page that will tell Google not to index the page. And is written as follows:<META NAME="ROBOTS" CONTENT="NOINDEX">
Verdict: Effective long term. This will keep your page from being indexed by the search engines if applied correctly
Nofollow
What is nofollow? - Depending on the context and where the nofollow directive is added. It can mean two things. It is either applied to individual links with the rel=”nofollow” attribute, or it can be applied in a meta robots directive <META NAME="ROBOTS" CONTENT="NOFOLLOW"> which tells Google not to follow any of the links on the page you’ve added it to.
Verdict: no, do not use this to remove an existing page from the index
If you are having issues with this or you have a more complex indexing problem that you need help with. Get in touch and we can help you.