---
title: "How to measure incrementality with a simple holdout test"
description: "Incrementality is the share of results your marketing caused. How to run a holdout on a small budget, read the result in margin and decide what to cut."
url: https://marginfix.ai/insights/how-to-measure-incrementality/
lastmod: 2026-10-12
---

[Home](/) › [Insights](/insights/) › How to measure incrementality with a simple holdout test

Measurement · 7 min read

# How to measure incrementality with a simple holdout test

![Portrait photograph of Cristian Bragau](/assets/founder/cristian-bragau-2026-10.jpg) **[Cristian Bragau](/about/cristian-bragau/)**, Founder & Lead Auditor · Reviewed Oct 2026 · verified against the base of 74 audits

The short answer

Run a holdout. Withhold one channel from a matched set of regions or a random share of customers, keep everything else the same, and compare the results with and without it. The difference is the incremental result. Non incremental paid spend appeared in 68% of our 74 audits, more than any other leak.

68% of audits found non incremental paid spend

54% of audits found attribution inflation

$22 of every $100 of audited spend was wasted paid spend

## Why the dashboards cannot answer it

Every ad platform can tell you which sales it touched. None of them can tell you, from inside its own reports, which of those sales would have happened anyway. That second question is [incrementality](/glossary/incrementality/), and the only way to answer it is to compare people who saw the spend with similar people who did not.

The gap matters most where the reported numbers look best. Branded search, retargeting, affiliate links and win back offers reach customers who are already on their way to buy, so they convert well and add little. The [attribution field guide](/insights/roas-attribution-field-guide/) covers why the platforms overcount; this guide covers the test that settles it. Non incremental paid spend is the most common [margin leak](/glossary/margin-leak/) in the corpus, found in 68% of the 74 audits.

## Choose a test your budget can run

Three designs cover almost every small business case. They differ in what they need and in how much they can prove.

TEST 01

Geo holdout

Pause the channel in a set of regions matched on past sales, keep it running everywhere else, and compare the two. The strongest design for channels you can switch off by location, such as paid social and search.

TEST 02

Audience holdout

Withhold the spend from a random share of the audience: a suppression list for retargeting, email or win back offers. Good wherever the audience is a list you control.

TEST 03

On and off

Switch the channel off for a period and compare with the periods around it. The cheapest test and the weakest, because the season, prices and other channels move at the same time. Use it to decide what deserves a real holdout.

Meta and Google Ads also run lift studies inside their own platforms for accounts that qualify. They are useful, but each platform measures its own campaigns. A geo holdout read against your own revenue measures the result you actually bank.

## Design a holdout you can trust

### Pick the groups before you look

Choose the holdout regions or the holdout share of customers before the test starts, and match them on past sales, not on hope. A holdout chosen after the results arrive can be made to say anything. For an audience test, a random split does the matching for you.

### Run it long enough

The test has to cover at least one full buying cycle, so a sale the channel started has time to finish. For most consumer brands that means several weeks; for a considered purchase or a B2B deal, longer. Keep launches, holidays and price changes out of the window, or record them and read the result with them in mind.

### Hold everything else still

Change one thing. If the holdout regions also get a new promotion, a new landing page or a different budget somewhere else, the test measures the mix and not the channel. Write down everything else that moved during the window.

### Decide the rule in advance

Before the test starts, write down which result means cut, which means cap and which means keep. A rule set after the numbers arrive tends to protect the channel everyone already liked.

## Read the result in margin, not in revenue

The arithmetic is short. Incremental orders are the orders in the exposed group minus the orders the matched holdout predicts for it. Divide the spend by the incremental orders and you have the incremental cost per order, the number to compare with what an order is worth to you after its variable costs.

The gap, worked through

Reported cost per order $20

Incremental cost per order $80

Arithmetic, not a case: retargeting is credited with 400 orders a month on $8,000 of spend, so the platform reports $20 an order. The holdout shows 300 of those orders would have happened anyway. 100 incremental orders on $8,000 is $80 an order, 4 times the reported cost.

Compare that $80 with the [contribution margin](/glossary/contribution-margin/) of an order. If an order keeps $50 after its variable costs, the reported $20 looks like a win and the real $80 loses $30 on every incremental order. When the test confirms a cut, [MER](/glossary/mer/) is the check that the saving was real: total revenue over total marketing spend should hold while the spend falls.

## What the published audits found with a holdout

Published audits where a holdout or a lift test decided the finding (8 of 74 cases)

| Where it was tested | The case | Annual figure |
| --- | --- | --- |
| Paid social against store sales | [Online ads were paying for in store demand](/cases/online-ads-were-paying-for-in-store-demand/) | $88k a year |
| An agency’s best reported campaign | [The agency’s ‘winning’ campaign was the biggest loser](/cases/the-agency-s-winning-campaign-was-the-biggest-loser/) | $53k a year |
| Affiliate and influencer links | [Affiliate commissions were being paid on sales the brand already owned](/cases/affiliate-commissions-were-being-paid-on-sales-the-brand-alr/) | $44k a year |
| A grocery loyalty scheme | [Loyalty points cost more than the repeat they drove](/cases/loyalty-points-cost-more-than-the-repeat-they-drove/) | $71k a year |
| An annual plan discount | [Annual plans were discounted below their retention value](/cases/annual-plans-were-discounted-below-their-retention-value/) | $57k a year |
| Win back offers | [Winback campaigns were rebuying customers who’d have returned free](/cases/win-back-campaigns-were-re-buying-customers-who-d-have-retur/) | $33k a year |
| Rep detailing and samples | [Detailing and sample spend produced no prescribing lift](/cases/detailing-and-sample-spend-produced-no-prescribing-lift/) | $64k a year |
| Congress and KOL sponsorship | [Congress and KOL sponsorship showed no measurable return](/cases/congress-and-kol-sponsorship-showed-no-measurable-return/) | $88k a year |

The test reaches beyond media. A loyalty scheme, a plan discount and a sponsorship each pay only if they change what customers do, and each of these audits found one that did not. Discounts face the same test: the [pricing audit](/insights/pricing-audit/) guide shows where they leak and how to withhold one and read the result.

## Where to test first

Start where the reported return is highest and the customer is closest to buying. Those channels collect the last touch on sales other channels started, and on sales nobody needed to start.

- **Branded search.** Customers searching your name were already looking for you. Test it against your organic ranking; the [Google Ads audit checklist](/google-ads-audit-checklist/) covers the setup. [Paid search was bidding on supply the platform already ranked for](/cases/paid-search-was-bidding-on-supply-the-platform-already-ranke/) is the marketplace version.

- **Retargeting.** Hold out a random share of the audience and compare their purchases with everyone else’s. [Paid was subsidizing a channel sales already owned](/cases/paid-was-subsidising-a-channel-sales-already-owned/) found brand terms and retargeting claiming deals that would have closed anyway.

- **Affiliate and influencer codes.** Compare buyers who used a link or a code with matched buyers who did not, and check how many already knew the brand.

- **Win back and loyalty offers.** Withhold the offer from a random share of lapsed or loyal customers. The ones who return anyway show what the offer never needed to buy.

Test one channel at a time, write down the result and the date, and test again when the channel’s budget or targeting changes. The [non incremental spend](/glossary/non-incremental-spend/) page has the definition and the cases behind it.

## Questions about measuring incrementality

### How long should an incrementality test run?

At least one full buying cycle, so sales the channel started have time to close, and longer for considered or B2B purchases. Keep launches, holidays and price changes out of the window, or record them and read the result with them in mind. [Incrementality, defined →](/glossary/incrementality/)

### Can a small business measure incrementality?

Yes. A geo holdout or an on and off test needs no special software: a channel you can switch off by region or by period, a few weeks and your own revenue data. Start with branded search or retargeting, where the gap between reported and incremental results is often widest. [The Google Ads audit checklist →](/google-ads-audit-checklist/)

### What is a good incremental result?

One where the incremental cost per order is below what an order keeps after its variable costs. There is no universal share to aim for: a channel can be 30% incremental and still pay if it is cheap enough, or 80% incremental and still lose money. [Contribution margin, defined →](/glossary/contribution-margin/)

[See how the audit tests incrementality →](/how-it-works/)

## More field notes

[Where SMB marketing budgets leak, and why nobody sees it, Benchmark report](/insights/where-smb-marketing-budgets-leak/) [Your ROAS is lying to you: an attribution field guide, Paid media](/insights/roas-attribution-field-guide/) [How to audit your agency without firing your agency, Working with agencies](/insights/how-to-audit-your-marketing-agency/) [What “good” looks like at $10k to $250k/month spend, Benchmarks](/insights/smb-marketing-spend-benchmarks/) [Pricing audit: where discounts eat your margin, Pricing](/insights/pricing-audit/)

[Request an audit →](/book/)
 [See the benchmark](/benchmark/)