A crawler, also called a spider or bot, is the automated program a search engine uses to discover and fetch web pages. It starts from a set of known URLs, downloads each page, extracts the links on it, and adds newly discovered URLs to a queue to fetch later, repeating the process continuously across the web.
Definition
A crawler, also called a spider or bot, is the automated program a search engine uses to discover and fetch web pages. It starts from a set of known URLs, downloads each page, extracts the links on it, and adds newly discovered URLs to a queue to fetch later, repeating the process continuously across the web.
The pages it fetches are then passed to indexing systems that parse the content, render it if needed, and decide what to store.
Crawlers respect a set of controls. robots.txt tells a crawler which paths it may or may not request. Meta robots and X-Robots-Tag directives on a fetched page tell it whether to index the page and follow its links.
Crawl-rate settings and the server's own response times govern how aggressively the crawler fetches. Sitemaps and internal links guide the crawler toward the URLs the site owner wants discovered.
In context
For a large iGaming affiliate site, crawler behaviour is a practical constraint. A site with tens of thousands of URLs — one review per operator, filtered and paginated list pages, tag archives, parameter variations — has a finite crawl budget, and if the crawler spends it on low-value parameter URLs and thin archives, it re-crawls important pages less often and may leave new content undiscovered for longer.
Managing this means blocking or noindexing the low-value URLs, keeping internal links pointing at the pages that matter, submitting a clean sitemap, and fixing slow responses and error pages that waste crawl attempts.
Crawlers also render JavaScript, but rendering is resource-intensive and can be delayed or incomplete, so content and links that only appear after client-side execution may be crawled late or missed. Affiliate sites that rely on JavaScript to build comparison tables, inject affiliate links, or load reviews should verify that the rendered HTML a crawler sees contains the content and links they depend on, using the search engine's own inspection tools rather than assuming parity with what a browser shows.
Worked example
An affiliate's server logs show that 60% of search-engine crawler requests in a month hit parameter URLs (?sort=, ?page=, ?filter=) and tag archives, while key reviews are re-crawled only every three weeks. After noindexing the archives and disallowing the sort parameters, crawler attention shifts to the reviews, which are re-crawled every few days and update in results faster.
Related terms
Frequently asked questions
Browse the full iGaming & affiliate glossary — hundreds of EN/RU terms with examples.
← Back to glossary