So, in various blog posts on SEO agencies’ websites, we all talk about link building, content marketing, and the introduction of AI until we are blue in the face.

But what about Googlebot?

Well, this article will talk about Robot.txt, when you give crawlers, such as Googlebot, instructions on how to crawl and index a company’s website.

 

Author: Ryan Walsh
Date: 03/03/2025
Subject: Technical SEO

 

What are search engine bots?

 

So, we always describe Googlebot and the other website crawlers as a bit like a bee that’s busy collecting nectar.

Googlebot crawls a website; it looks for changes, improvements, and new pages and then reports back to Google HQ.

Therefore, the crawler, i.e., Googlebot, spots SEO changes to a page and when a business adds a new piece of content marketing.

Yet websites have a “crawl budget”, meaning there’s only so much time that Google and the other search engines can spend crawling and indexing a website. Businesses must use their crawl budget wisely, asking Googlebot only to crawl the pages they want to appear in Google’s SERP’s. This is where the robots are. Text syntax comes in; it can look like a shopping list, but it should contain a list of pages you indexed and ones you do not, such as the WordPress login screen.

 

What Is Robots.txt?

So, it doesn’t sound very easy, but a robot.txt file is a plain text document. It should then set out a list of instructions for the search engine crawlers, which some SEO companies call bots, and others call spiders, but they are the same thing. As mentioned earlier, all they do is set out which pages you want to be crawled and indexed by Googlebot and which you don’t.

What does Robots.txt do for SEO?

It helps you manage your crawl budget.

Even Google, with its vast, colossal crawling and indexing abilities through Googlebot, can’t spend forever and a day indexing a website.

So, it has to have a crawl budget; this budget is how much time the robots, such as Googlebot, can spend crawling and indexing that site. Businesses of all sizes, from sole traders to PLC, have to use their crawl budget wisely to make sure that the content marketing you want to be crawled gets crawled and the less important pages, such as the WP admin login screen, don’t get indexed, for example.

Set it up correctly and hire an expert.

So, a word to the wise: carefully set up your robot.txt instructions.

 

Why is this so important?

It’s a bit like giving Google and the other search engines a building floor plan and explaining which doors can be accessed and which are locked.

But, if you do something silly, like lock all the doors or mistakenly set up no-index instructions across the site, there’s nothing to index.

Which means your rankings can drop off a cliff.

It’s basically like de-indexing your website, and it ultimately disappears from Google altogether.

 

User-Agent Directive

What is a “User-Agent-Directive”?

User-agent directive: This line of text indicates which spiders and web crawlers the instruction applies to. For example, you could specify a user agent, such as Googlebot.

 

Sitemap Directive

What is a sitemap directive?

The sitemap is simply an instruction to point the crawlers (Googlebot) at your business XML sitemap.

 

What does a robot.txt file look like?

A sitemap resembles a shopping receipt, yet you often disallow or allow directives. These will make up most of the instructions. Then you have elements such as sitemap or sometimes crawl delay instructions. However, many leading SEO companies believe that Googlebot ignores crawl delay instructions.

 

Why do you need robots.txt?

Is a robot.txt file

So, without Google or other search engine bots crawling and indexing a company’s website, it would never appear in search engine results.

Sometimes, a CMS can have so many pages and posts that without a robot.txt, they will never get indexed.

So, a robot.txt file makes it super simple for Googlebot or another bot to have a list of pages the company wants to be indexed.
And this is important, as imagine writing, say, a blog post that takes over a day to complete. You have incorporated the ideas and opinions of many staff members within your company.

And then, you conduct some research and some relevant facts.

So, you’ve spent a lot of time producing this work. It will never appear in Google’s SERPs if it isn’t indexed.

If it doesn’t appear in Google’s SERP, it won’t get clicks; you won’t make more sales if it doesn’t.
So, a robot.txt, if it’s regularly updated, makes the Googlebot job much more straightforward, and instantly, it will know which pages should get indexed and which shouldn’t.

 

How do you know if you have a robots.txt file?

Add/robot.txt to the end of the website’s URL / domain name.

Then, if there’s a robot.txt and sitemap in place, the page should detail the URL where the sitemap is kept.

 

Is adding a sitemap difficult?

This is an absolute piece of cake if you have a WordPress website. Some free WordPress plugins exist, such as Yoast SEO or All in One SEO. Both are good and without coding knowledge, you can set up a site map in minutes.

 

Will setting up a sitemap mean that my website will get every page indexed?

Setting up a sitemap is the first step in helping Googlebot to know which pages or blog posts should be indexed.

However, we would also recommend setting up a Google Search Console account. Then, submit your sitemap for indexing. This tells Google that you have a sitemap in place and that you should index that sitemap.

 

Can internal linking help improve indexation?

It most certainly can, so, for example, most SEO experts would agree that it’s good to keep an essential, up-to-date sitemap. Then, set some directives, such as which pages you want indexed and which you don’t. On top of that, internal linking can help Google discover new pages and blog posts.

Internal linking has many advantages, but here are the main ones that spread link juice, and the other is helping to accelerate the indexation of new work.

So, Googlebot, for example, may return to an old blog post to see if any changes have been made to it. From that, it may spot anchor text and a new internal link, which it follows, and then finds that you have written a new page, so it indexes that, too. So, because Google uses links to crawl and index websites, adding internal links can also help Googlebot to find new pages.

 

Do all websites have the same crawl budget?

We would say no.

Leading SEO agencies would also agree.

The reason is that your top-ranked websites are in the top 4; well, they typically get indexed more often than lower-ranking pages.

So, for example, a business that’s ranked on page 10 of Google SERPs, say a brand new company, well, their crawl budget is much more likely to be less than a business which occupies the number 1 position in Google’s SERPs.

Therefore, businesses that rank high on Google typically have a much higher crawl budget.

This is why you must use your crawl budget wisely, especially for your start-up business in Cardiff.

 

Can you speed up the indexation of pages manually?

You can undoubtedly set up a Google Search Console account; when you write a new blog post, you can submit it manually for Googlebot to crawl and index the page.

Therefore, when a copywriter has finished writing a page, instead of waiting days, or sometimes weeks, for Googlebot to crawl and index that page, you can manually submit it to Google. Then, it will be put in a queue for Google to crawl and index that page.

 

Which pages should be no-indexed?

The pages that should not be indexed are those your business doesn’t want to appear in Google’s search engine results pages.

For example, it could be your company’s terms and conditions page or the CMS login for Drupal or WordPress.

It could be because you have duplicated pages and want one page indexed and the others not.

How do you go about creating the perfect Robots.txt File? We offer step-by-step Instructions:

So, the first thing to do is consult an SEO agency. If the instructions are set up incorrectly, you could accidentally stop the homepage from getting indexed, which would be disastrous to any business’s search engine optimisation.

However, you could help your SEO agency by doing the following “heavy lifting”- you could work out the pages you want indexed and the ones you don’t. Please ensure all the main pages, such as the homepage, about us page, and service pages, are included in the list of pages you want to be indexed. Then, you could set these up using a simple Microsoft Notepad file; then, you can ask your chosen SEO company to make them into instructions on whether the page should get indexed.

Then, you can set instructions for the search engines you want to optimise your website. For example, Google, Microsoft Bing, Yandex, etc.

Common Robots.txt mistakes and how to avoid them

In our view, setting up the robots.txt document is crucial for the SEO’s success moving forward.
So, take your time and ask your SEO consultant for help. If this is set up incorrectly, it could lead to the main pages not getting indexed, which would be disastrous for your company’s SEO.

 

How to test if your Robots.txt file is working?

You can use Google Search Console to test whether your robot.txt is working. However, as mentioned earlier, we strongly recommend hiring an SEO consultant to ensure it’s written correctly.

 

When “Disallowing a page”, which pages should we decide to block in the Robots.txt?

This will vary from business to business; however, generally speaking, it’s the pages you don’t want to appear in the SERP’s. A good example is the CMS content management systems login page, which is only used by staff or your appointed SEO agency.

 

How our agency can help

We are one of the longest-established agencies in Cardiff. We can help local and national companies improve every local or organic search engine optimisation aspect, from setting up robot.txt files to building good quality links.

The business is managed by Ryan Walsh, who has over 10 years of SEO experience. During this time, we have worked with many different companies. We can help improve your technical SEO by writing content marketing that allows you to draw in more organic traffic.

Call Walsh & Partners today.