Robots.txt
Robots.txt is a plain text file on your website that tells search engine crawlers which pages and directories they are allowed or not allowed to access.
Robots.txt is a text file placed at the root of your website that tells search engine crawlers which parts of the site they are allowed to access and which they should skip.
What Robots.txt Means in Marketing
Every website can have a robots.txt file at the path yourdomain.com/robots.txt. When a crawler like Googlebot visits your site, it checks this file before crawling anything else. The file lists rules that instruct which bots can go where.
Robots.txt is a courtesy file, not a security mechanism. Well-behaved crawlers like Googlebot follow the rules it contains. Poorly behaved bots or scrapers may ignore it entirely. For this reason, it is the wrong tool for protecting sensitive information: if you want something inaccessible, you need authentication or a server-level block, not robots.txt.
The practical use of robots.txt in crawlability management is telling search engines to skip low-value pages: admin sections, internal search results, duplicate paginated content or staging directories. This prevents crawl budget from being spent on pages that should never rank.
How Robots.txt Works
The file uses a simple directive format. A User-agent line specifies which bot the rules apply to. Disallow lines list paths the bot should not access. Allow lines can override specific paths within a disallowed directory. A wildcard asterisk in the User-agent field covers all bots.
A typical configuration might block crawlers from accessing /admin/ and /internal-search/ while allowing everything else. More targeted configurations can block specific bots entirely, which is useful if you want to prevent content scrapers from consuming server bandwidth.
One important limitation: disallowing a URL in robots.txt prevents Googlebot from reading its content, but does not prevent the URL from appearing in search results. If other sites link to a disallowed page, Google can discover the URL from those links and show it in results as a link without a description. If you want a page excluded from results, use a noindex directive instead of, or in addition to, robots.txt.
Robots.txt Example
An e-commerce site disallows its filtered product pages, the URLs created when users sort by price or filter by colour. These pages have near-identical content to the main category pages and create thousands of near-duplicate URLs. By blocking them in robots.txt, the site prevents Googlebot from spending crawl budget on duplicates and ensures crawling stays focused on the canonical product category pages that actually need to rank.
Why Robots.txt Matters for Marketers
Robots.txt is one of the first places to check during a technical SEO audit. A misconfigured disallow rule is one of the most common causes of an entire section of a site disappearing from search results. Keeping the file simple, accurate and regularly reviewed ensures that your most important content is always accessible to crawlers, while low-value or duplicate pages do not consume the crawl budget those bots have allocated to your site.
Frequently Asked Questions
Does robots.txt prevent pages from being indexed?
No, and this is a critical distinction. Robots.txt tells crawlers not to access a page, but if the page has been linked to externally, Google can still index it using data from those external sources without ever reading the page's content. If you want a page to not appear in search results, you need a noindex tag on the page itself, not just a robots.txt disallow. Using robots.txt to hide sensitive content is therefore unreliable.
What is the difference between robots.txt and a noindex tag?
Robots.txt works at the crawl stage: it tells bots whether to access the URL at all. A noindex tag works at the indexing stage: the bot accesses the page, reads the tag and decides not to store it in the index. If you block a page in robots.txt, Googlebot cannot read the noindex tag because it never accesses the page. To remove a page from search results reliably, use a noindex tag rather than robots.txt alone, because robots.txt cannot guarantee the URL won't be indexed from external links.
Can you accidentally block important pages with robots.txt?
Yes, and it happens regularly. A common scenario is a developer adding a disallow rule to block a staging environment and either applying it too broadly or forgetting to update it before a production launch. A single overly broad disallow rule can block hundreds or thousands of pages at once. Since search engines respect robots.txt before accessing any content, affected pages disappear from search results without any visible error on the page itself.
Want this handled on your own site? See my SEO services. Or see all my services.
Need help putting this concept into practice on your campaigns? Share your current targets and I will suggest the right approach.