How to measure incrementality with a simple holdout test
Cristian Bragau, Founder & Lead Auditor · Reviewed · verified against the base of 74 auditsRun a holdout. Withhold one channel from a matched set of regions or a random share of customers, keep everything else the same, and compare the results with and without it. The difference is the incremental result. Non incremental paid spend appeared in 68% of our 74 audits, more than any other leak.
Why the dashboards cannot answer it
Every ad platform can tell you which sales it touched. None of them can tell you, from inside its own reports, which of those sales would have happened anyway. That second question is incrementality, and the only way to answer it is to compare people who saw the spend with similar people who did not.
The gap matters most where the reported numbers look best. Branded search, retargeting, affiliate links and win back offers reach customers who are already on their way to buy, so they convert well and add little. The attribution field guide covers why the platforms overcount; this guide covers the test that settles it. Non incremental paid spend is the most common margin leak in the corpus, found in 68% of the 74 audits.
Choose a test your budget can run
Three designs cover almost every small business case. They differ in what they need and in how much they can prove.
Pause the channel in a set of regions matched on past sales, keep it running everywhere else, and compare the two. The strongest design for channels you can switch off by location, such as paid social and search.
Withhold the spend from a random share of the audience: a suppression list for retargeting, email or win back offers. Good wherever the audience is a list you control.
Switch the channel off for a period and compare with the periods around it. The cheapest test and the weakest, because the season, prices and other channels move at the same time. Use it to decide what deserves a real holdout.
Meta and Google Ads also run lift studies inside their own platforms for accounts that qualify. They are useful, but each platform measures its own campaigns. A geo holdout read against your own revenue measures the result you actually bank.
Design a holdout you can trust
Pick the groups before you look
Choose the holdout regions or the holdout share of customers before the test starts, and match them on past sales, not on hope. A holdout chosen after the results arrive can be made to say anything. For an audience test, a random split does the matching for you.
Run it long enough
The test has to cover at least one full buying cycle, so a sale the channel started has time to finish. For most consumer brands that means several weeks; for a considered purchase or a B2B deal, longer. Keep launches, holidays and price changes out of the window, or record them and read the result with them in mind.
Hold everything else still
Change one thing. If the holdout regions also get a new promotion, a new landing page or a different budget somewhere else, the test measures the mix and not the channel. Write down everything else that moved during the window.
Decide the rule in advance
Before the test starts, write down which result means cut, which means cap and which means keep. A rule set after the numbers arrive tends to protect the channel everyone already liked.
Read the result in margin, not in revenue
The arithmetic is short. Incremental orders are the orders in the exposed group minus the orders the matched holdout predicts for it. Divide the spend by the incremental orders and you have the incremental cost per order, the number to compare with what an order is worth to you after its variable costs.
Compare that $80 with the contribution margin of an order. If an order keeps $50 after its variable costs, the reported $20 looks like a win and the real $80 loses $30 on every incremental order. When the test confirms a cut, MER is the check that the saving was real: total revenue over total marketing spend should hold while the spend falls.
What the published audits found with a holdout
| Where it was tested | The case | Annual figure |
|---|---|---|
| Paid social against store sales | Online ads were paying for in store demand | $88k a year |
| An agency’s best reported campaign | The agency’s ‘winning’ campaign was the biggest loser | $53k a year |
| Affiliate and influencer links | Affiliate commissions were being paid on sales the brand already owned | $44k a year |
| A grocery loyalty scheme | Loyalty points cost more than the repeat they drove | $71k a year |
| An annual plan discount | Annual plans were discounted below their retention value | $57k a year |
| Win back offers | Winback campaigns were rebuying customers who’d have returned free | $33k a year |
| Rep detailing and samples | Detailing and sample spend produced no prescribing lift | $64k a year |
| Congress and KOL sponsorship | Congress and KOL sponsorship showed no measurable return | $88k a year |
The test reaches beyond media. A loyalty scheme, a plan discount and a sponsorship each pay only if they change what customers do, and each of these audits found one that did not. Discounts face the same test: the pricing audit guide shows where they leak and how to withhold one and read the result.
Where to test first
Start where the reported return is highest and the customer is closest to buying. Those channels collect the last touch on sales other channels started, and on sales nobody needed to start.
- Branded search. Customers searching your name were already looking for you. Test it against your organic ranking; the Google Ads audit checklist covers the setup. Paid search was bidding on supply the platform already ranked for is the marketplace version.
- Retargeting. Hold out a random share of the audience and compare their purchases with everyone else’s. Paid was subsidizing a channel sales already owned found brand terms and retargeting claiming deals that would have closed anyway.
- Affiliate and influencer codes. Compare buyers who used a link or a code with matched buyers who did not, and check how many already knew the brand.
- Win back and loyalty offers. Withhold the offer from a random share of lapsed or loyal customers. The ones who return anyway show what the offer never needed to buy.
Test one channel at a time, write down the result and the date, and test again when the channel’s budget or targeting changes. The non incremental spend page has the definition and the cases behind it.
Questions about measuring incrementality
How long should an incrementality test run?
At least one full buying cycle, so sales the channel started have time to close, and longer for considered or B2B purchases. Keep launches, holidays and price changes out of the window, or record them and read the result with them in mind. Incrementality, defined →
Can a small business measure incrementality?
Yes. A geo holdout or an on and off test needs no special software: a channel you can switch off by region or by period, a few weeks and your own revenue data. Start with branded search or retargeting, where the gap between reported and incremental results is often widest. The Google Ads audit checklist →
What is a good incremental result?
One where the incremental cost per order is below what an order keeps after its variable costs. There is no universal share to aim for: a channel can be 30% incremental and still pay if it is cheap enough, or 80% incremental and still lose money. Contribution margin, defined →