How does a holdout group work?
Before a campaign or journey launches, a random slice of the eligible audience is assigned to the holdout. Those users match the recipients in every way except one: they never receive the message. After the campaign runs, you compare conversion rates between the two groups, and the difference is the incremental lift — the conversions the campaign actually caused. Open and click rates flatter every send; only a holdout separates the conversions you caused from the conversions that were already coming.
What is the difference between a global and a per-campaign holdout?
A per-campaign holdout is excluded from one specific message, so it measures that message's lift — useful for expensive sends like a win-back sequence. A global holdout is excluded from all marketing for a longer window, often a quarter, and measures what your whole program is worth. Braze's global control group, for example, lets you reserve between 1% and 15% of your user base and reports uplift in sessions and conversions against it. Per-campaign holdouts answer tactical questions; a global holdout tells you whether messaging moves your North Star metric at all.
Why it matters for a two-person team
Small teams cannot afford to keep running campaigns on faith. A holdout is cheap insurance against that: most journey builders can carve one out with an A/B split node where one branch simply exits without sending — the guide to journey orchestration covers how the ab_split node works. Reading the lift at the end is the kind of call a CRO specialist makes: keep the campaign, rewrite it, or kill it and reclaim the send budget.
How big should a holdout group be?
Big enough to detect the lift you care about, small enough that the opportunity cost stays acceptable. On small audiences, percentages get noisy fast: a 5% holdout of 500 users is 25 people, and a single conversion swings the result. Common practice among messaging platforms is single-digit percentages for global holdouts — Braze caps its global control group at 15% — and somewhat larger slices for one-off campaign tests. When in doubt, prefer one global holdout read over months to many tiny per-campaign holdouts read once.