In a site audit of a small media site recently, I found that of the 500 content pages that it has, Google had only indexed five or so pages.
This seemed strange, so upon checking the code, we found the following embedded near the top of every page: <meta name=”robots” content=”noindex, nofollow”/>
This is called the meta robots tag, and the ‘noindex, nofollow’ command basically tells Google not to include the site in Google’s index and not to follow any links on the page.
Because this particular site had the meta robots tag on the top of every page on the site, none of it was getting any search traffic as Google had ignored the entire site.
So what is the meta robots tag used for? In some cases, there are pages that you wouldn’t want Google to include in their index, such as a privacy notice or a terms and conditions page. Using this tag will omit those pages from Google.
What you need to do is a quick site audit of your own site to make sure that your content is being indexed properly.
You can first check what Google thinks your site looks like and then view the source code to make sure that your meta robots tag is not misconfigured.
You can typically do this by right clicking on a blank spot on the page, click on “View Page Source” and look out for the above meta robots tag.
You can help us (and help yourself)
Small and medium businesses and startups have never needed credible, independent journalism and information more than now.
That’s our job at SmartCompany: to keep you informed with the news, interviews and analysis you need to manage your way through this unprecedented crisis.
Now, there’s a way you can help us keep doing this: by becoming a SmartCompany supporter.
Even a small contribution will help us to keep doing the journalism that keeps Australia’s entrepreneurs informed.