SEO & Digital Marketing
What is Robots.txt?
Also called robots.txt, robots file, robots exclusion protocol.
Short definition
Robots.txt is a text file at the root of a domain that tells search engine crawlers which parts of a website they may and may not crawl; the file controls crawling, not indexing.
How it works
This file always sits at a fixed address, for example yourdomain.com/robots.txt. It contains groups of rules: User-agent names the crawler being addressed, Disallow names the paths that may not be crawled, and Allow creates exceptions. The sitemap location can also be listed here. Crawlers that follow the standard read it before crawling, while malicious crawlers can simply ignore it.
It does not keep pages out of the index
This is the most common misunderstanding. Google makes clear that robots.txt is not a way to keep pages out of search results. A blocked page can still be indexed if it is linked from elsewhere, and it then appears without a description. To prevent indexing, use noindex or protect the page with a password. Noindex only works if the page is not blocked by robots.txt, because Google has to be able to read the page to see that rule.
It is not a security tool either
Anyone can read robots.txt. Listing the address of an admin panel or a secret folder there actually tells outsiders where it is. Sensitive pages are protected with login and access settings on the server. Another classic mistake: a site-wide Disallow rule carried over from the staging server to the production website. Every time a new website goes live, open this file once to make sure it is correct.
Related terms
Resources
- Google Search Central: Introduction to robots.txt (developers.google.com)
- RFC 9309: Robots Exclusion Protocol (rfc-editor.org)