A holdout test is a controlled experiment in which a randomly selected group — the holdout or control — is deliberately excluded from a marketing activity so its behaviour can be compared with an equivalent exposed group.
Definition
A holdout test is a controlled experiment in which a randomly selected group — the holdout or control — is deliberately excluded from a marketing activity so its behaviour can be compared with an equivalent exposed group. The difference in outcomes between the two, given random assignment and adequate size, is a causal estimate of what the activity actually produced.
Holdouts are the practical mechanism behind incrementality measurement.
They come in several forms: audience holdouts (a share of eligible users never receive a retargeting or CRM campaign), geographic holdouts (matched regions where a channel is paused), and time-based or staggered rollouts (regions or segments switched on in sequence). The key requirements are random or well-matched assignment, a group large enough to detect the expected effect, a clean measurement window, and discipline not to contaminate the holdout by reaching it through another channel.
In context
iGaming operators use holdouts to price marketing truthfully. A CRM holdout — a random slice of the base that receives no promotional emails, push or SMS for a period — reveals how much of reactivation and retention revenue the CRM programme actually drives versus what players would have done anyway, and it also serves as an ethical baseline, showing whether promotional pressure is increasing spend among at-risk players.
A retargeting audience holdout shows the incremental value of chasing users who already visited. A geo holdout on brand paid search shows how much of that traffic organic would have captured for free.
For affiliate decisions, a holdout can be run at the channel or partner level: pause a specific affiliate or a group of similar affiliates in matched regions, or withhold a cohort from the bonus that a particular partner's traffic relies on, and measure the true incremental registrations and deposits. This is how an operator distinguishes affiliates who bring genuinely new players from those who intercept existing demand, and it produces evidence for renegotiating rates in either direction.
Holdouts cost some certain short-term revenue for uncertain long-term insight, so they are used selectively on the biggest spend questions, and their results are treated as the tie-breaker when attribution models disagree.
Worked example
An operator holds 10% of its active base out of all CRM promotions for six weeks. The holdout's revenue is 12% lower than the messaged group — the real incremental effect of CRM — far less than last-touch reporting credited.
The holdout also shows no rise in responsible-gambling flags versus the messaged group, which had a small increase, prompting a review of promotion frequency.
Related terms
Frequently asked questions
Browse the full iGaming & affiliate glossary — hundreds of EN/RU terms with examples.
← Back to glossary