The short answer
A robots.txt file is a plain text file at the root of your website that tells search engine crawlers which URLs they can and cannot request. In our view, most small business websites need only a short file that allows normal crawling, keeps unimportant URLs out of crawl traffic, and points to a sitemap.
A robots.txt file is a set of instructions for automated web crawlers, such as Googlebot, telling them which URLs on your website they can and cannot request. According to Google Search Central, its main purpose is to manage crawler traffic so your server is not overloaded and crawlers do not waste time on unimportant or repetitive pages, such as internal site search results.
In our view, most small business websites do not need complex rules. A short, clear file is usually enough to let search engines crawl your main pages and find your sitemap.
How robots.txt works and where it lives
Google’s guide to creating a robots.txt file sets out a few strict rules on format and placement. The file must be plain text encoded in UTF-8, and Google imposes a file size limit of 500 KiB.
It also has to live at the root of your host, such as https://www.example.com/robots.txt. Any robots.txt file placed inside a subdirectory is ignored, and a robots.txt file only applies to the specific host or subdomain where it is placed.
Inside the file, rules are built using four main directives.
User-agentnames the crawler the rule applies to, with an asterisk (*) meaning all crawlers that read the file.Disallowtells that crawler not to request URLs matching a specific path.Allowoverrides a broaderDisallowrule so a specific URL or subfolder can still be crawled.Sitemapgives crawlers the full web address of your XML sitemap.
A worked example for a small business website
In our view, a typical small business website with a blog, a handful of service pages, and an internal search function only needs a few lines. An illustrative file might look like this.
User-agent: *applies the rules below to all crawlers.Disallow: /searchstops crawlers from requesting internal search result pages.Allow: /confirms that the rest of the site is open to crawl.Sitemap: https://www.example.com/sitemap.xmlpoints crawlers directly to your sitemap.
In our view, it is best to keep your robots.txt as simple as possible. Every extra Disallow line adds a risk of accidentally blocking a page or folder you actually want customers to find.
What not to put in robots.txt
The biggest misconception about robots.txt is that it hides pages from Google. As Google explains, robots.txt controls crawling, not indexing.
Blocking a URL in robots.txt does not guarantee it will stay out of Google Search. If other pages link to that URL, Google can still index the address itself without showing a page description.
Because of that, you should never use robots.txt to hide private or sensitive data. Google recommends using a noindex directive or password protection instead.
There is an important catch if you use noindex alongside robots.txt. Googlebot has to crawl a page to read a noindex tag, so if you block that page in robots.txt, Google will never see the noindex instruction.
In our view, you should also avoid blocking CSS, JavaScript, or image files that search engines need to render your layout properly.
How to check your file in Google Search Console
Once your file is live, you can verify how Google reads it using the robots.txt report in Google Search Console, found under Settings and then robots.txt.
The report shows the files Google found for up to your top 20 hosts, along with the timestamp of the last crawl and any syntax warnings or fetch issues. If you update your file after fixing a mistake or launching a new site, you can also use the report to request a recrawl.
If you would like a second pair of eyes on your technical setup, our SEO audit includes technical and indexing checks across your key pages.
Sources
Where a claim depends on how Google or another company says something works, we link to their own words so you can check it.
- Introduction to robots.txt, Google Search Central. Supports what robots.txt controls and its limits on indexing and privacy.
- Create and submit a robots.txt file, Google Search Central. Supports file placement at the host root, UTF-8 plain text format, the 500 KiB limit, and the four main directives.
- robots.txt report, Google Search Console Help. Supports how to check your file in Search Console, see crawl timestamps and syntax warnings, and request a recrawl.
About this answer. We are an independent studio and this is our own research, experience and opinion, shared as general guidance. It is not legal, financial or other professional advice, and what works for one business may not work for another. Where the law or a regulator is involved, check the official guidance for your situation or ask a qualified adviser. Our sources are listed above.