Incrementality vs. attribution
Attribution divides credit for a conversion among the touchpoints that preceded it — last click, first touch, some weighted curve. It answers "which of our activities was nearby when this happened?" Incrementality asks the harder, more useful question: "would this have happened anyway?" A branded-search click gets full credit under most attribution models and often near-zero incrementality, because the buyer was already coming. The two numbers can disagree by an order of magnitude, and the incremental one is the one the spend decision should follow.
Revenue incrementality, done properly
For website personalization the incrementality unit that matters is dollars per visitor: revenue in treated sessions versus held-out sessions. Conversion rate alone can mislead — an experience can convert more visitors into smaller orders, lifting CVR while revenue falls — so the revenue verdict has to be measured directly and decomposed into conversion rate times order value.
The statistics need care too. Session revenue is zero-inflated and heavy-tailed: most sessions are worth nothing and rare large orders dominate the mean, which breaks the assumptions behind a t-test. AXO uses a seeded percentile bootstrap — thousands of resamples, deterministic so the same inputs always yield the same confidence interval — and only reports significance when the interval's lower bound clears zero.
How to run an incrementality test on website personalization
The design is a permanent holdout, not a one-off study. Randomly assign a fixed share of sessions — AXO uses roughly one in ten — to a control that sees no personalization anywhere on the site. Keep the assignment deterministic (hash the session) so a visitor never flips between arms mid-visit, and let both arms run concurrently, so seasonality, campaigns, and traffic-mix shifts wash out of the comparison instead of masquerading as lift.
Then read two numbers, not one: conversion-rate lift, and revenue per visitor with its confidence interval. Declare nothing below minimum sample size — a verdict that arrives early is a verdict you'll retract. Because the holdout never stops running, the test tracks the program as it changes: every new variant, segment, and season is measured against the same standing control, which is what makes the resulting number an incrementality claim rather than a launch-week anecdote.