Technical SEO / Indexation Issues

Local SEO & E-commerce Indexing Issue Support

You could be writing and publishing the best content marketing in the world on any particular product or service. But if search engines like Google, Bing, or even

Yandex can’t read it (crawl and index it), it won’t appear in their search engine results.

For SEO boffins, this is what we call a “page indexing issue”.

It can happen for a multitude of reasons, from an incorrectly configured robots.txt to the page being an orphan page that Googlebot can’t find.

Visibility in business marketing is everything.

Simply put, if the page can’t be read, it can’t be found; if it can’t be found, it stands no chance of appearing in the SERPs (search engine results pages).

We’ve seen this firsthand: previous agencies wrote articles until they turned blue from exhaustion. They thought this would be the ticket to higher organic rankings on Google.

However, if Googlebot is prevented from crawling and indexing, that work is about as useful as a chocolate teapot! This was because the “no-index” tab was left on in the CMS! This meant no articles were indexed, even though they had been working on a content marketing strategy and implementation for the last 6 months!

We have the expertise to spot problems such as no XML sitemap in place, poor internal linking, wasted crawl budgets, and thin content pages.

It could also be because someone used AI as their writing tool, which has now caused those pages to be de-indexed. Plus, if you operate in a YMYL (your money, your life) business sector, the content marketing needs to be supremely high quality.

Google’s algorithm is like a car engine; it needs quality fuel to run right.

The same goes for content marketing: if it’s not written to a good standard, if there’s duplication, or if there’s mass use of AI, the page won’t rank well on Google.

Indexing issues: what are they, and how do you solve them?

The search engines need to serve quality answers to questions. This keeps customers coming back. If SERPs don’t help customers by providing relevant results or answers, shoppers go elsewhere. Google’s purpose is therefore simple: serve the best answer to any question it’s asked.

Therefore, built into the algorithm are a multitude of ways of preventing spam. We have seen the Google Panda and Penguin updates rock businesses’ rankings in the SERPs.

As a result, pages get pulled from the index, or never make Google’s index in the first place.

Here’s the best way we know how to explain this: it doesn’t matter what you sell, luxury chocolate or hyper cars online; the pages need to offer quality advice (Google EEAT).

Now, Google and its almighty brain, called its algorithm, sometimes have millions of pages it can serve as an answer.

There are often only 10 spaces in the organic results.

There are often only 3 results in the Google Business results

And in terms of AI, Google AI Overviews only serve just one answer

So, competition is fierce. If you don’t supply a good-quality answer, then the competition will.

This is why so many pages don’t make the cut; this often results in the page being described in Google Search Console as “crawled and not indexed”.

This basically means Google took a look and doesn’t believe it’s worth indexing because the quality isn’t high enough.

There could be many reasons this happened; however, common ones include duplicate content, AI-generated content, or thin content.

Google E-E-A-T / YMYL reviews, using Google Search Console to monitor content performance

The definition of insanity is doing the same thing and expecting different results.

So, if you are surrounded by competitors ranking in top positions on Google across a range of keywords, yet you’re not, something needs to change.

Perhaps the pages are content-thin, too much AI text has been used, or perhaps the information is just, well, not cutting it.

Therefore, Google rewards quality; it rewards hard work.

So, if you fire up that espresso maker and start writing the best advice in the business, you will be considered the best in the business.

It takes time to rank higher on Google. You have to improve your EEAT score over a sustained period of time.

This means showing off your level of experience and expertise on that subject.

Then the SEO boffins you employ have to improve the website’s authoritativeness and trustworthiness.

Which means building solid links, not spammy links.

It means gaining business reviews and getting your business mentioned and cited on respected websites. This improves semantic SEO, which means Google’s algorithm and Knowledge Graph start to understand your brand and that you have been cited on relevant, high-quality websites.

You’re basically becoming a big cheese in your industry.

This takes time; it takes sweat; it takes commitment.

So the agencies promising the world, for very little in the way of expenditure, well, they are telling porky pies. In fact, their noses may be growing as they make those promises.

De-indexing is on the rise due to Google algorithm updates

Google has a hard job on its hands.

Imagine how many AI posts get published per day.

Imagine how many half-baked blog posts, with zero chance of making their way to page one, get indexed per day.

Then there are copywriters in every country working for various SEO agencies that get told to bash out 1000-word articles in a very limited amount of time.

The result: rubbish work that doesn’t serve anyone.

It doesn’t help the business rank higher, as Google’s algorithm will filter it out.

It’s just another page that’s crawled and not indexed in Google Search Console.

Then we have the AI content, where the same advice is slapped on thousands of other websites.

So, these pages often don’t get indexed, or if they are, they can get de-indexed.

This is what the countless Google updates are for.

They used to have funky names, such as Google Penguin updates; however, now they are increasingly just named after the date they were released.

And Google rolls out hundreds of these each year.

So, what does this have to do with indexing issues? Well, everything.

It’s why a business can rank on Google one moment, then vanish into thin air the next.

When a particularly large Google Core Update rolls out, Google has the opportunity to really examine the content marketing.

To see, well, is it high quality or not?

If it’s not, if it turns out someone has been rehashing AI content all over the shop, those pages can then get deindexed.

Addressing page indexing issues because of quality issues

Businesses today want to appear in Google’s organic results, the AI overviews and on the LLM’s such as ChatGPT.

However, for this to happen, the content marketing quality needs to be very high.

You can’t therefore publish content marketing that’s been written in a rush.

Neither can you use AI content.

The reason is, when a business starts to add too much, well let’s be frank rubbish blog posts, this will affect the entire websites SEO performance.

This is often referred to as the pages quality score.

This is assessed through Google EEAT for example.

What some of the common reasons why pages can’t get indexed:

Server errors can cause indexing issues, this are often referred to as “Server error (5xx)”.

For example, we have worked on e-commerce businesses, where the website was going down because of hosting issues.

If the website is down, it can’t get indexed by Googlebot.

So, if the website is down for any period of time, this can lead to a drop in organic rankings.

Redirect error

Redirect errors can put Googlebot in a spin quite literally. For example, you may have a product that’s no longer stocked, then you want to redirect the shopper to the newer and better version of that product. However, if you create a redirect loop, causing Googlebot to get confused, well then you are simply wasting your businesses crawl budget.

URL blocked by robots.txt

Sometimes web designers and web developers write quickly into the robot.txt not to crawl and index say a new page they are working on.

However, sometimes, even when the page is then live, they forget to remove that instruction.

Which results in well, that page can’t appear in Google’s SERP’s.

Its not therefore visible, so it cant sell the products or services that the page is advertising.

Soft 404

A 404 error could be caused for a whole range of reasons, however, if you have too many of these, well your definitely going to be causing the websites bounce rate to increase.

Not found (404)

A page could be 404 because it’s been deleted.

This causes a negative user experience.

The reason being, someone may have wanted to buy an item from you. Therefore the shopper can select the meta title out of all the other options.

But instead of seeing the item they wanted to buy, they get greeted with 404 page not found error.

This often means the shopper “pogo sticks” meaning they leave, and go to a competitor’s website instead.

This increases the companies bounce rate.

Crawled – currently not indexed

Within your Google Search Console account you may have seen “crawled currently not indexed”. This is not because Googlebot couldn’t be bothered getting around to indexing the work, it often means the work is not high enough quality to index.

Again that brings us back to our point that we said where Google’s algorithm has sometimes millions of pages it can use to rank on page one.

Therefore, there’s no point in indexing more, if the quality is not high enough.

However, if you really put your back into, write a piece that’s truly the best, well then that page will get indexed.

Duplicate without user-selected canonical

Duplication issues are a headache for customers and Google.

Imagine a pair of trainers, they are listed on the website on different pages.

The descriptions might well be the same, they are just in different product categories.

This means Google often doesn’t know which is the main page.

Plus, shoppers can sometimes get lost in maze of products which haven’t been sorted into categories.

Duplication damages SEO, therefore avoid having duplicated pages.

Page with redirect

Pages with redirects can work well, if implemented correctly.

For example, a product may have ranked 1st organically. This could have brought in a ton of organic visitors.

However, one small problem, the manufacturer then discontinued making that item- so what should you do.

Well this is where re-directs could be useful, directing customers to a new page, selling the newer version.

However, if you have “redirect loops” where Googlebot goes in circles, so does the customer, eventually resulting in them leaving say the e-commerce shop, well this is not good SEO.

Unstable Rankings Because of Indexing Issues

Unstable organic rankings can be caused by all manner of things. For example, the websites hosting could be going on and off more than the kitchen light. Which means sometimes Googlebot can crawl the work, on other occasions, well it can’t.

Other reasons could be because of “thin content” pages, where the CMS basically has a lot of pages with not much text.

Alternatively, the text just doesn’t offer much in the way of useful advice.

Then there’s cannibalisation issues, this happens a lot!

Its basically when blog posts or main pages are competing with themselves.

For example, you’re selling a smoothie maker, so you want to rank for “smoothie makers + brand”- but you have say 7 product pages all selling the same thing.

This causes cannibalisation, Google simply doesn’t know which page to place in its SERP’s.

Sometimes it’s made to decide which it thinks is the main product page.

However, because you have cannibalisation issues, and perhaps a direct competitor doesn’t, well guess which business ranks higher?

Yes, the competitor site, simply because they may have poured all their time and effort into creating one Brillant page crammed with helpful buying advice. And you on the other hand have 7 pages all selling the same blender!

Why choose Walsh & Partners?

We are quite simply the SEO experts.

We don’t just help you to gain more sales and leads.

We also use the best methods, which means we like to steady the ship.

Some agencies cut corners, they may for example use AI to write the text, or just make a whole lot of duplicated pages.

However, we like to implement a well thought out SEO strategy. One where we get to know the business, plus who runs it. Then we want a good understanding of where you want to take the company. For example, is there a product line you want to push, and perhaps not so much on some other products or services.

Then armed with this knowledge, we can create a plan of action.

This means, knowing which competitors are the competitors we essentially need to beat.

We then look at their backlinks.

We then write better buying guides.

We then sort out all of the technical SEO glitches, such as robot.txt telling Googlebot not to crawl and index a page, when really that’s a key selling page! So it needs to be indexed.

We are the SEO experts.

With 14 years’ experience, we are simply one of the best SEO companies on the planet!