Most personalization "lift" is a model grading its own homework. Ours is a randomized holdout, a z-test, and a scorecard that would rather say nothing than flatter you.
We built a reporting screen that refuses to render. If a site doesn't have enough traffic for the math to mean anything, the AI Scorecard doesn't show a cautious verdict — it doesn't show up at all. That's a strange product decision, so here's the reasoning. It's really an argument about how this industry measures.
The standard personalization dashboard always says yes. It compares conversions among people who saw the campaign against last month, or against a "model-estimated baseline," or against nothing. The number goes up because your traffic mix changed, or the season did, or another campaign landed — and the tool takes credit, because the tool graded its own homework. We've written about the questions that expose this; this is what the honest version looks like from the inside.
AXO's answer is old-fashioned: a controlled experiment, always on. Roughly one visit in ten is held out — chosen deterministically by hashing the session, so a visitor doesn't flip between arms mid-visit — and never sees personalization. Treatment is compared against holdout with a standard two-proportion z-test. That gap is the lift, and it's the only number in the product allowed to call itself that.
The honesty rules follow from the math. Below a minimum count of treated and held-out sessions, the verdict reads learning — not winning. Below a traffic floor where statistical significance is out of reach no matter how long you wait, the scorecard hides itself entirely, rather than dangle a permanent "still learning" that reads like a promise. And when live holdout data is too thin and we show a projection replayed from history instead, it's labeled projected in the interface — not in a footnote. The significance threshold is a fixed constant in the code, so nobody can quietly loosen it until noise looks like a win.
Conversion rate alone can lie to you in one specific, expensive way: personalization that nudges more people to buy smaller things. Conversion up, average order value down, revenue flat or negative — and a conversion scorecard applauds the whole way down.
So revenue gets its own verdict: incremental dollars per visitor, treatment against holdout, with every session counted exactly once so a multi-zone page can't double-count an order. Revenue is zero-inflated and heavy-tailed — most sessions are worth $0 and one whale can bend an average — so a t-test's assumptions don't hold. We bootstrap instead: thousands of resamples with a seeded generator, meaning the same inputs produce the same confidence interval every single time. The result only reads significant when the interval's lower bound clears zero, and it's decomposed into conversion rate times order value so you can see which lever actually moved. What that lift is worth is then arithmetic, not faith.
The measurement is the product philosophy. The holdout is on by default, the tests are standard statistics rather than a proprietary attribution model, projections are labeled, and the scorecard would rather show you nothing than flatter you. If a number can't survive those rules, we don't think you should be making decisions on it either — ours included.
With a randomized holdout: a slice of traffic — in AXO, roughly one visit in ten, assigned deterministically per session — that never sees personalization, running at the same time as treated traffic. Comparing conversion between the two groups with a standard two-proportion z-test isolates the lift the personalization caused. Before/after comparisons and model-estimated baselines cannot do this, because traffic mix and seasonality move the number on their own.
Personalization can raise conversion rate while lowering average order value — nudging more people to buy smaller things — leaving revenue flat or negative while a conversion-only dashboard reports a win. The fix is measuring revenue per visitor against the holdout directly, decomposed into conversion rate times order value, so both levers are visible at once.
Because on low-traffic sites, statistical significance is mathematically out of reach, and showing a permanent "still learning" verdict reads like a promise that a verdict is coming. Below a traffic floor, AXO hides the scorecard entirely rather than imply precision it cannot deliver. Showing nothing is more honest than showing noise.