What does “Crawled but currently not indexed mean in Google Search Console?

Date: 20/02/2024
Subject: Google Search Console / Technical SEO
Author: Ryan Walsh

We think Google Search Console or GSC is fantastic, but why do we believe this?

Well, it’s a tool used by the most prominent companies globally. Equally, a self-employed person just setting up could use it also. It is brilliant; it’s a dashboard, a window into how your website is performing, and it tells you important metrics such as the following:

 

– Which keywords your website is ranking for in Google’s index
– It tells you impression information
– How many clicks
– Has Google imposed a Manual Action Penalty
– Backlinks information, such as which websites are linking and how many
– And much, much more

 

Yet there is also an important section which tells you which pages are indexed and which are not.

This article explains more about indexed and unindexed pages and what this means for a company’s SEO.

What does this “Crawled – Currently Not Indexed” in GSC mean?

So this description—“Crawled, currently not indexed”—appears in Google Console. It is the same as saying Google knows that the page or blog post is there; Google has crawled it. Yet, it hasn’t been added to Google’s index for many reasons.

 

What is Googlebot?

Googlebot is a bot that never stops crawling and indexing the web. It’s a bit like a bee, going from flower to flower, collecting the nectar, and then returning to the hive.

However, Google moves from website to website, reading the pages and determining which ones should be added to Google’s index.

 

But why would a page not be indexed?

There are many different reasons why any page cannot be indexed. It could be something as simple as perhaps the whole website has a “no-index” switch on the CMS settings. In that case, you flip the switch and submit the site to Google Search Console for indexation.

On the other hand, it could be technical issues or just Google’s algorithm that has decided not to index the page. For example, it could be because the page has thin content and lacks quality.

 

 

Space junk

Create a picture of content marketing pieces, pages and blog posts floating in outer space, with planet Earth in the image.

So, what do we call “space junk” here at this agency?

Every day, millions upon millions of content marketing pieces are written and published on websites.

Some are of exceptional quality; it’s clear an expert has spent some considerable time offering quality advice.
On the other hand, content marketing could be what we call “space junk”—rubbish work floating around, offering no quality advice, and just cluttering up the place.

Don’t produce space junk; instead, do the following:

 

Create content marketing that has the following qualities:

– Pour your heart and soul into it, show why you’re the bee’s knees, the expert in your particular business sector
– Take your time to do a good job
– AI, don’t use it just yet; instead, get an expert to create the work
– Appy good on-page SEO
– Offer quality advice

 

And the result?

If the work is high quality, it will most likely be indexed. So, more pages can be indexed in your Google Search Console index report. Yet that’s, of course, if all the technical elements are fixed.

Let’s explain more about how “crawling and indexing works” and what Googlebot does.

 

Crawling

Crawling is automated. Different search engines use different bots, which are assigned different names. For example, Google uses something called “Googlebot,” which jumps from website to website, crawling and reading the page.

These bots are often called crawlers or spiders. Their job is to go from website to website, crawling and looking for changes. For example, you might have written a long article. It’s brand new, so you will want to appear in Google’s results, yet before that happens, it needs to get crawled and indexed.

 

What does indexing mean?

Google keeps a massive index, a bit like a vast library. It places a page or blog post in the index according to its quality.

So, Google has an index or a Google Knowledge Vault. This is where a page’s ranking factors are calculated; the page with the most substantial ranking factors and Google E-EAT is then ranked at the top of the SERPs.

 

Sound complicated?

It is because there are more than 200 other ranking factors, an overarching framework of Google E-EAT, and machine learning. All these different elements are used to calculate a page’s quality.

Some SEO companies in Cardiff may say this is the DA score for a website.

That’s the domain authority—however, that calculation is a bit simplistic. It’s good; you can use it to understand whether a website or page is quality. However, DA is a bit simplistic.

Instead, when you factor in machine learning, Google RankBrain, bounce rates, dwell time, ranking factors, and semantic SEO, you need to use not simplistic SEO metrics but the real brain of a top SEO consultant.
Why?

Because there are so many elements to decipher, is that page of good quality rubbish, or what needs to be improved?
Like a mechanic who is new at what they do, they may follow the instructions of a fault code reader. However, a good mechanic, a top mechanic, knows my sight and listening, what is happening to an engine, and often pin-point problems just by listening to an engine run.

Like a good SEO consultant, very few in any one city know how to properly weigh up how good and evil a business’s SEO is.
Ranking

So, we explained a bit about how Google decides where to rank a website in its search engine results pages. This is based on the company’s ranking factors and how strong they are.

Yet Google’s algorithm also uses “relevance,” “authority,” and “user experience.” These are also part of Google E-EAT. Basically, in terms of on-page and off-page SEO, SEO needs to be very strong. You also need a well-designed website offering a good user experience if you want to keep the bounce rate low.

It just doesn’t cut it.

 

 

Today’s content marketing must be very high-quality to rank on page one of Google. It needs to have good on-page and off-page SEO. The

Google E-EAT score needs to be through the roof. Therefore, the bar is set very high.

For example, it doesn’t matter if you offer advice on how to fix a leaking tap or legal advice because you run a solicitor’s practice. The pages that get ranked on page one of Google usually are deemed pages or blog posts that are better than thousands, if not sometimes millions, of other pages talking about the same title.

So, the bar is set high.

When we started our SEO agency, getting a business on page one was much more straightforward. This is because the competition for the keyword is lower. In any given city, a page might only have 100 other pages it had to beat to get on page one of Google. Now, that same business could compete with 10x the number of pages to get on page one of Google.

The level of competition has increased, so a page must be of superior quality to be ranked on Google.

 

Technical issues

There could be several technical issues holding your SEO back.

For example, the hosting could be unreliable, and there could have been several server errors. If Googlebot cannot crawl and index the page because of server errors, the website could drop in the SERPs.

It could be because the robots.txt has not been set up correctly, now giving instructions not to index a page when you might want that page, for example, to be indexed. This could be because, let’s say, the page was being redesigned, yet the designer forgot to remove the no-index instruction.

 

The Google Helpful Content Update

As we come onto a bit later in this article, it’s something called Google’s crawl budget.

Even though Google has incredible computing power, it has to work out and dedicate computing power to where it can best be used.

Otherwise, with the sheer number of pages published daily, Google couldn’t handle all the work.

Therefore, Google allocated a crawl budget to each site. This means Google will only spend so much time crawling and indexing a website, so you don’t want to waste your crawl budget with Googlebot routinely crawling, say, the CMS login screen.

You can help Googlebot by writing an XML sitemap, which sets out which pages should be indexed and which shouldn’t. For example, you could set the XML sitemap to not index the WordPress login screen, as that’s not useful to a customer.

This is crucial, as it lets you keep your crawl budget for the pages you want indexed.

In “Google’s Helpful Content Update, ” Google clearly stated that it just wants helpful pages with a high Google E-EAT score in its index.

So, low-quality, spammy, or AI pages will not get indexed; Google wants quality.

To produce pages with high Google E-EAT, you must put much time and effort into creating that work. So keep your business crawl budget for publishing high-quality pages and blog posts.

Therefore, instead of publishing a piece of content marketing once a week, that’s just content thin or AI work. Instead, invest your time and energy into creating better-quality work.

For example, create a content calendar focused on publishing high-quality work once a month.

 

Avoid duplicate content marketing.

We would avoid duplication issues, as these can damage your organic SEO.

When discussing duplication issues, we mean copying text internally and avoiding copying text from other websites.

Many businesses have duplication issues, such as 1,000 product descriptions, where the whole page has text copied from the supplier’s website. Instead, the text must be unique to your business, so offer your unique product description.

 

Does this take a massive amount of time?

Sure, it does.

However, the largest e-commerce businesses and global websites know that the text must be unique for on-page SEO, so there can’t be any duplication issues.

 

5xx server errors

You could have the best web designers in the world design your company website, yet if the hosting is unreliable, this could affect the business’s search engine optimisation.

What we mean by that is that for some businesses, Googlebot may crawl and index that page daily. If the website is down, perhaps for an extended period or multiple days, this can cause problems with the SEO because Google will often spot that the website is not live.

This means that Google will not want to send shoppers to a page not found 404 error or a 5xx server issue.

This causes a negative user experience, as the shopper can’t reach the page they want.

Unreliable hosting has two main issues. It causes poor UX because customers can’t reach their desired product or service page. From a technical point of view, it’s also bad for SEO simply because Googlebot can’t crawl and index the page.

Therefore, it’s crucial that any server issues are fixed. If the hosting is unreliable and slow, then this needs to be fixed. We recommend switching hosting companies and asking your web designers for help.

 

Site structure

The website needs to be very well designed. Think about page layout, ensuring each product category and service has a dedicated page. Add internal links. Make sure that your business’s XML site map is up to date. We would also recommend adding breadcrumb navigation, as this helps a shopper navigate the website better, for example, if they wish to return to a page.

 

New content marketing

For those SEO experts out there who have been offering organic SEO services for some time, they will know about something called the

Google Freshness update.

Google doesn’t like a website to get style.

It is like websites and businesses which update their websites regularly.

However, don’t think it’s just a matter of quick content marketing after you have an espresso and write 750 words of waffle.

Google knows if a business is writing a marketing waffle or if the page is high quality.

So, the point we are making here is simple: Google rewards businesses with higher rankings if the work being added regularly is of good quality.

If the website is drowned in complete marketing waffle and content-thin pages, this work will not help improve the company’s SEO. It can lower the quality score, the Google E-EAT score, so that this work can damage the business’s organic search engine optimisation.

Quality is the name of the game.

So, you need to add quality work reasonably regularly.

 

Mobile indexing

We all know that more and more people are buying goods and services off their smartphones than ever before.

So, when we think about indexation, can a page be indexed? We must also consider whether the mobile version can be indexed. There could be a glitch, and the mobile version cannot be indexed. This could mean a drop in organic traffic, but if the mobile version of the website cannot be indexed, then it won’t appear in the SERPs.

 

Internal linking

We highly recommend linking different pages using internal links. Of course, this needs to be done in a white-hat way.

However, internal links offer two different things. First, they help teleport a shopper to a different page. So, for example, if you’re selling a chainsaw, you could offer an internal link to the replacement chain page, where they may wish to purchase a separate item.

So, you can help your business sell more products by offering internal links.

Internal links are also good because they help Googlebot crawl and index different pages. For example, Googlebot can use an internal link to follow it and potentially find a blog post that has not been indexed yet.

Then, Googlebot can index the page and rank it in the SERP.

Is there anything that we could do to speed up indexing?

Yes, there most definitely is. When you publish a new page, you can open your Google Search Console and submit it for indexing.

This is your way of asking Google to index that page rather than having to wait, potentially sometimes multiple weeks, for Googlebot to find, crawl, and index the page.