Crawl budget is the number of URLs a search engine's crawler will fetch on a site within a given period.
Definition
Crawl budget is the number of URLs a search engine's crawler will fetch on a site within a given period. It is a function of crawl capacity (how much the crawler can request without overloading the server, based on response times and errors) and crawl demand (how much the search engine wants to crawl the site, based on its size, freshness and perceived importance).
For small sites it is rarely a constraint; for large sites it determines how quickly new content is discovered and how often important pages are re-crawled.
When a large site spends its crawl budget on low-value URLs — parameter combinations, faceted filters, session URLs, infinite pagination, thin archives — it re-crawls its important pages less often, and new or updated content takes longer to appear or update in results. Managing crawl budget means steering the crawler toward the pages that matter and away from the ones that do not.
In context
On a large iGaming affiliate site, crawl-budget waste is usually visible in server logs: a large share of crawler requests hitting ?sort=, ?page=, ?filter= URLs, tag and author archives, and internal search results, while the core reviews and guides are fetched only every few weeks. The levers to fix it are the standard ones — robots.txt disallow for parameter and search paths, noindex for thin archives (which also, over time, reduces crawl demand for them), a clean sitemap listing only canonical indexable URLs, strong internal linking to important pages, and fixing slow responses and error pages that consume requests without producing indexable content.
The payoff of concentrating crawl budget is faster indexing of new content, more frequent re-crawls of updated pages (so a changed bonus or a new operator review reflects in results sooner), and cleaner index-coverage reports because the crawler is no longer generating "crawled — not indexed" noise from low-value URLs. It is a technical hygiene task rather than a growth tactic on its own, but on a site with tens of thousands of URLs it removes a real ceiling on how responsive the site can be in search.
Worked example
Server-log analysis of an affiliate site shows 65% of monthly crawler requests going to parameter URLs and tag archives. After disallowing the parameters and noindexing the archives, crawler requests shift to content pages, new reviews are indexed within two days instead of two weeks, and "crawled — not indexed" URLs in the console drop by 8,000.
Related terms
Frequently asked questions
Browse the full iGaming & affiliate glossary — hundreds of EN/RU terms with examples.
← Back to glossary