Is this the solution to your Magento indexing problem?

I’m talking to you today about the Google index. We know mobile-first is coming, and it’s being rolled out as we speak. Watch for an announcement about it in your Google Search Console. This, by the way, is just awesome for finding out things about the index that you didn’t previously know. We were doing a search for one particular client and found a whole bunch of issues that were previously hidden.

One of the main problems we had was trying to get their site indexed properly. When we looked at their Search Console we could see that they had thousands of errors popping up, including over 24,000 soft 404s. Now, a soft 404 is basically when Google is thinking you want your server to respond with a 404 error page, but what you’ve actually done is a redirect.

In this situation, the Googlebot was being redirected and interpreting that as a soft 404, which is a big problem. After some digging we found that people weren’t being redirected, just the Googlebot, and only the main Googlebot that’s attached to the main desktop. To make matters worse, this is exclusively a Magento problem. We worked on isolating the trouble (which was a bit of a trick) and found that it’s a user validation issue. We tried using the mobile Googlebot on our Magento site, and it was redirected just as it should have been. Even more surprising, that seemed to fix the problem, and the desktop Googlebot is now behaving itself!

Now, this is an intermittent problem that’s not even hitting all Magento sites all the time. But for now, if you’ve got a site that’s having indexing problems, switch off user agent verification. That should get things back where they should be, and your traffic and revenue should rise quickly after that.

This article originally published on

NOW READ: Can removing old content help your SEO?


Notify of
1 Comment
Newest Most Voted
Inline Feedbacks
View all comments
Lee Goldsworthy
Lee Goldsworthy
4 years ago

For anyone looking for where to configure this, it’s in the doco here:

Be sure to check if you actually do have this issue before changing your user agent validation settings, as this can be useful in other ways.

SmartCompany Plus

Sign in

To connect a sign in method the email must match the one on your SmartCompany Plus account.
Or use your email
Forgot your password?

Want some assistance?

Contact us on: or call the hotline: +61 (03) 8623 9900.