robots.txt is a text file placed at the root of a domain that tells web crawlers which paths on the site they are allowed or not allowed to request. It uses a simple syntax of User-agent lines (which crawler the rules apply to) and Allow / Disallow lines (which URL paths).
Definition
robots.txt is a text file placed at the root of a domain that tells web crawlers which paths on the site they are allowed or not allowed to request. It uses a simple syntax of User-agent lines (which crawler the rules apply to) and Allow / Disallow lines (which URL paths).
Compliant crawlers, including the major search engines, read it before crawling and respect its directives; it is a convention, not an enforced security control.
The most important thing to understand about robots.txt is what it does not do. Disallowing a URL prevents crawling, but it does not prevent indexing: if other pages link to a disallowed URL, a search engine can still list it in results as a bare link with no snippet, because it knows the URL exists but was not allowed to fetch its content.
To keep a page out of the index, the correct tool is a noindex directive on the page itself — which requires that the page be crawlable, so it must not also be disallowed in robots.txt.
In context
On a large iGaming affiliate site, robots.txt is used to keep crawlers away from paths that have no value and would waste crawl budget: internal search results, faceted-filter parameter combinations beyond a whitelist, session or tracking parameters, admin and staging paths, and infinite-scroll or calendar URL patterns. Done well, this concentrates crawler attention on the content pages that should rank.
The common, costly mistakes are disallowing something that should be crawled — an accidental Disallow: / left from a staging config, blocking a CSS or JS path the page needs to render (which breaks mobile-first rendering), or blocking a section whose pages carry noindex, so the noindex is never read and the URLs linger in the index as bare links. robots.txt should be reviewed after every site migration and template change, tested with the search engine's robots.txt tester, and paired correctly with noindex and canonical rather than used as a substitute for them.
Worked example
An affiliate audits its robots.txt and finds Disallow: /reviews/ left over from a redesign, which had quietly de-crawled 800 review pages carrying noindex,follow. It removes the disallow so the pages can be re-crawled, the noindex is finally read, and the intended crawl and index behaviour is restored.
Related terms
Frequently asked questions
Browse the full iGaming & affiliate glossary — hundreds of EN/RU terms with examples.
← Back to glossary