Why a holdout is the only honest baseline
Every alternative baseline is confounded. Before/after comparisons absorb seasonality, traffic-mix shifts, and whatever else launched that month. "Model-estimated baselines" are a vendor's model estimating the vendor's own contribution. A concurrent, randomly assigned control group is the one design where the only systematic difference between the groups is the treatment itself — which is why clinical trials, ad-platform lift studies, and honest personalization measurement all converge on the same shape.
The practical tell when evaluating a tool: ask to run a holdout. A confident vendor helps you set one up. A nervous one explains why holdouts are unnecessary, or hard, or "already baked in."
How AXO runs holdouts
The holdout is always on, not a special study. Roughly one visit in ten is assigned to control by hashing the session — deterministic, so a visitor never flips between arms mid-visit — and that session sees no personalization anywhere on the site. Treatment and holdout conversion rates are compared with a standard two-proportion z-test.
Two honesty rules sit on top. Below minimum sample sizes the verdict reads learning, not winning. And below a traffic floor where significance is mathematically out of reach, the scorecard hides itself entirely rather than imply a verdict is coming.