Skip to main content
Legia

SEO & Digital Marketing

What is Robots.txt?

Also called robots.txt, robots file, robots exclusion protocol.

Short definition

Robots.txt is a text file at the root of a domain that tells search engine crawlers which parts of a website they may and may not crawl; the file controls crawling, not indexing.

How it works

This file always sits at a fixed address, for example yourdomain.com/robots.txt. It contains groups of rules: User-agent names the crawler being addressed, Disallow names the paths that may not be crawled, and Allow creates exceptions. The sitemap location can also be listed here. Crawlers that follow the standard read it before crawling, while malicious crawlers can simply ignore it.

It does not keep pages out of the index

This is the most common misunderstanding. Google makes clear that robots.txt is not a way to keep pages out of search results. A blocked page can still be indexed if it is linked from elsewhere, and it then appears without a description. To prevent indexing, use noindex or protect the page with a password. Noindex only works if the page is not blocked by robots.txt, because Google has to be able to read the page to see that rule.

It is not a security tool either

Anyone can read robots.txt. Listing the address of an admin panel or a secret folder there actually tells outsiders where it is. Sensitive pages are protected with login and access settings on the server. Another classic mistake: a site-wide Disallow rule carried over from the staging server to the production website. Every time a new website goes live, open this file once to make sure it is correct.

Related Legia service SEO Technical fixes, content structure, and local search, measured through to incoming inquiries.