What is robots.txt, and what should be in it?

Answers · More Than Normal, Canterbury

The short answer

A robots.txt file is a plain text file at the root of your website that tells search engine crawlers which URLs they can and cannot request. In our view, most small business websites need only a short file that allows normal crawling, keeps unimportant URLs out of crawl traffic, and points to a sitemap.

A robots.txt file is a set of instructions for automated web crawlers, such as Googlebot, telling them which URLs on your website they can and cannot request. According to Google Search Central, its main purpose is to manage crawler traffic so your server is not overloaded and crawlers do not waste time on unimportant or repetitive pages, such as internal site search results.

In our view, most small business websites do not need complex rules. A short, clear file is usually enough to let search engines crawl your main pages and find your sitemap.

How robots.txt works and where it lives

Google’s guide to creating a robots.txt file sets out a few strict rules on format and placement. The file must be plain text encoded in UTF-8, and Google imposes a file size limit of 500 KiB.

It also has to live at the root of your host, such as https://www.example.com/robots.txt. Any robots.txt file placed inside a subdirectory is ignored, and a robots.txt file only applies to the specific host or subdomain where it is placed.

Inside the file, rules are built using four main directives.

  • User-agent names the crawler the rule applies to, with an asterisk (*) meaning all crawlers that read the file.
  • Disallow tells that crawler not to request URLs matching a specific path.
  • Allow overrides a broader Disallow rule so a specific URL or subfolder can still be crawled.
  • Sitemap gives crawlers the full web address of your XML sitemap.

A worked example for a small business website

In our view, a typical small business website with a blog, a handful of service pages, and an internal search function only needs a few lines. An illustrative file might look like this.

  • User-agent: * applies the rules below to all crawlers.
  • Disallow: /search stops crawlers from requesting internal search result pages.
  • Allow: / confirms that the rest of the site is open to crawl.
  • Sitemap: https://www.example.com/sitemap.xml points crawlers directly to your sitemap.

In our view, it is best to keep your robots.txt as simple as possible. Every extra Disallow line adds a risk of accidentally blocking a page or folder you actually want customers to find.

What not to put in robots.txt

The biggest misconception about robots.txt is that it hides pages from Google. As Google explains, robots.txt controls crawling, not indexing.

Blocking a URL in robots.txt does not guarantee it will stay out of Google Search. If other pages link to that URL, Google can still index the address itself without showing a page description.

Because of that, you should never use robots.txt to hide private or sensitive data. Google recommends using a noindex directive or password protection instead.

There is an important catch if you use noindex alongside robots.txt. Googlebot has to crawl a page to read a noindex tag, so if you block that page in robots.txt, Google will never see the noindex instruction.

In our view, you should also avoid blocking CSS, JavaScript, or image files that search engines need to render your layout properly.

How to check your file in Google Search Console

Once your file is live, you can verify how Google reads it using the robots.txt report in Google Search Console, found under Settings and then robots.txt.

The report shows the files Google found for up to your top 20 hosts, along with the timestamp of the last crawl and any syntax warnings or fetch issues. If you update your file after fixing a mistake or launching a new site, you can also use the report to request a recrawl.

If you would like a second pair of eyes on your technical setup, our SEO audit includes technical and indexing checks across your key pages.

Sources

Where a claim depends on how Google or another company says something works, we link to their own words so you can check it.

  1. Introduction to robots.txt, Google Search Central. Supports what robots.txt controls and its limits on indexing and privacy.
  2. Create and submit a robots.txt file, Google Search Central. Supports file placement at the host root, UTF-8 plain text format, the 500 KiB limit, and the four main directives.
  3. robots.txt report, Google Search Console Help. Supports how to check your file in Search Console, see crawl timestamps and syntax warnings, and request a recrawl.

About this answer. We are an independent studio and this is our own research, experience and opinion, shared as general guidance. It is not legal, financial or other professional advice, and what works for one business may not work for another. Where the law or a regulator is involved, check the official guidance for your situation or ask a qualified adviser. Our sources are listed above.

Get our new answers by email

One short email a week with the answers we've published. Unsubscribe with one click.

We email you a link to confirm first. See our privacy notice.

Want to know where your website stands

A written SEO audit shows what to fix first, in order of priority, for £450.

Get an SEO auditBook a discovery call

No obligation, and we reply by email.