Matched market testing vs geo holdout is mostly a question of direction and selection. Are you adding media or taking it away? And are your test markets hand-matched to controls, or drawn at random? Get those two answers and the choice usually makes itself.
One warning first: the labels are used loosely. Vendors call nearly any market-level experiment a “geo test,” and some call every holdout a matched market test because the markets were matched. This guide uses the terms the way most in-house teams do, and spends more time on the trade-offs than the vocabulary.
The short answer
| Matched market test | Geo holdout | |
|---|---|---|
| Typical question | Does this new channel, or more spend, add customers? | Is this always-on channel still adding customers? |
| What changes | Media added or increased in test markets | Media paused in holdout markets |
| How markets are picked | Paired by history, often 1–4 test markets | Random draw, often 20–30% of markets |
| Markets you need | Works with 6–15 | Better with 15+ |
| Cost during the test | Extra spend | Lost sales in held-out markets |
| Biggest risk | A twin drifts for its own reasons | Unlucky draw, or leakage into holdouts |
| Answer it gives | Value of the added dollars | Value of the dollars you already spend |
How a matched market test works
You start from history. Take six to twelve months of weekly outcomes for every market (guests, car counts, new patients) and find pairs or small groups that moved together. Not the same size: the same shape. Then, within each pair, pick the test market. Ideally by coin flip, because the market you would have picked by hand is the one you secretly expect to win.
The test markets get the new channel or the extra spend. The matched controls carry on unchanged. After six to eight weeks, you compare each test market with its controls, or better, with a weighted blend of all the steady markets fitted on the pre-period.
Google researchers have published a structured version of this: a search for matched groups that respects an advertiser’s constraints, designed for cases where pure randomization breaks down because markets differ sharply or are few (Source: Au, Google Research, 2018). That is a fair description of most multi-location brands.
Where matched market tests shine
- Launching a channel you’ve never bought: CTV, radio, Google Local Services Ads, a new social platform.
- Pricing a budget increase before committing it across all markets.
- Brands with too few markets for a random draw to balance, which is most brands under 15 markets.
How a geo holdout works
A geo holdout goes the other way. A channel is running everywhere; you switch it off in a set of markets and watch what happens to their locations compared with the markets where it stays on. If outcomes in the dark markets slip relative to their comparison, the channel was doing real work there.
With enough markets, you should pick the holdouts at random, stratified so big and small markets land on both sides. Google’s own geo experiment papers lean on randomization for exactly this reason: it protects against hidden differences between the groups that nobody thought to match on (Source: Vaver & Koehler, Google Research, 2011). Their later Trimmed Match design pairs geos first and randomizes within pairs, which is a sensible middle ground (Source: Chen, Longfils & Remy, Google Research, 2021).
Where geo holdouts shine
- Checking the channels that carry the most budget: paid social, non-brand search, direct mail.
- Settling the “brand search would have come anyway” argument.
- Brands with 15 or more markets that can spare a quarter of them for two months.
What each test costs: a worked example
Illustrative numbers. A 60-location auto service chain spends about $20,800 a week on paid search and Meta across 20 markets. Each shop does roughly 260 cars a week, and gross margin on an average $85 ticket is about 50%, so a car is worth $42.50. The weekly gap between any test group and its comparison wanders by about 4% in the pre-period. The team is weighing two tests.
| Geo holdout | Matched market heavy-up | |
|---|---|---|
| Markets in test | 5 of 20 (15 shops), random | 4 of 20 (12 shops), matched pairs |
| Cars per week in test shops | 15 × 260 = 3,900 | 12 × 260 = 3,120 |
| Length | 8 weeks | 6 weeks |
| Minimum detectable effect | 2.8 × 4% ÷ √8 ≈ 4.0% | 2.8 × 4% ÷ √6 ≈ 4.6% |
| Spend change | −$5,200 / week (25% of budget) | +$4,300 / week |
| Why that size | Planned holdout share | 4.6% × 3,120 = 143 cars at a guessed $30 each |
| If the media works | 1,248 cars lost × $42.50 = $53,040 margin lost, minus $41,600 saved: net cost $11,440 | 858 cars gained × $42.50 = $36,465, minus $25,800 spent: net gain $10,665 |
| If the media does nothing | Net gain: $41,600 saved | Net cost: $25,800 spent |
Look at the shape of that table. The holdout is cheapest exactly when the channel is useless, which is the scenario you’re trying to rule out. The heavy-up is cheapest when the channel works. So a CFO who suspects paid search is mostly brand traffic should love the holdout, and a CMO who is confident in a new channel should prefer the heavy-up.
One more detail hiding in the numbers. The team guessed the channel drives about 4% of car counts. A six-week holdout would have an MDE of 4.6%, so it would probably miss a real 4% effect. Eight weeks brings the MDE to about 4.0%. That is why the holdout runs longer.
Which test should you run?
There is one case that trumps the chart. If regional operators will quietly add their own promotions in any market you take dark, run the heavy-up. A holdout contaminated by a rescue BOGO is worse than no test.
And if you have fewer than about eight markets, neither design will give a tight answer. A longer matched market test with a synthetic blend still beats guessing, but expect a wide range and plan to repeat it.
Under the hood, the analysis is the same
Whatever you call the design, the read should be the same: for each test market, build a weighted blend of steady markets that tracked it during the pre-period, and measure the gap afterwards. Then run placebo checks, pretending each comparison market was the test, to see how big a gap noise alone produces. A single “twin city” is too fragile; one local news story can sink it.
Randomization and matching are not rivals either. Matching makes the groups similar; randomization within matched pairs stops you from fooling yourself about which market got the media. Do both when you can. For the full mechanics of holdout design (buffers, spillover, test length), see our guide to geo holdout tests for multi-location brands.
Common mistakes with both designs
- Matching on size instead of shape. Two markets with the same revenue but different seasonality are bad twins. Match on how the weekly series moved.
- Hand-picking test markets. Even in a matched design, flip a coin within each pair.
- Holdouts that aren’t dark. Radius targeting, national campaigns and Performance Max can keep serving in held-out markets. Check delivery by region weekly.
- Heavy-ups too small to see. A 10% spend increase almost never moves outcomes enough to measure. Size from the MDE, not from what feels safe.
- Reading a heavy-up as the channel’s average return. It tells you what the next dollars buy. That’s usually less than the average dollar.
- Forgetting the tail. Customers keep arriving after the media changes. Read three to four weeks past the end, especially for holdouts, where the decline can lag.
Questions people ask about matched market tests and geo holdouts
What is a matched market test?
A matched market test changes advertising in a few markets and compares them with markets whose past sales moved the same way. The difference after the change estimates the effect of the advertising. It’s common for new channels and spend increases, and it suits brands with too few markets for a fully random design.
What is a geo holdout test?
A geo holdout switches a running channel off in a set of markets, chosen at random where possible, while it continues elsewhere. If outcomes at locations in the dark markets fall relative to their comparison, the channel was driving customers there. It’s the cleanest way to check whether an always-on channel still pays.
Is a matched market test less accurate than a geo holdout?
Not inherently. Accuracy depends on pre-period fit, number of markets, test length and whether anything else changed. A matched test with a good weighted blend can be very precise. Its weakness is selection bias if markets are picked by hand; randomizing within matched pairs fixes most of that.
How many markets do I need for a geo holdout?
Enough that a random draw gives balanced groups and a solid comparison pool: roughly 15 or more markets, holding out 20 to 30% of them. With fewer, match markets into pairs and randomize within each pair, or run a matched market heavy-up instead.
Can I run both at once?
Not on the same channel in the same markets. You can run a holdout on one channel and a heavy-up on another if the market groups don’t overlap and each has its own clean comparison pool. In practice most brands run one test at a time and alternate through the year.
Pick the design that answers the decision in front of you, price it honestly, and run it long enough to see the effect you care about.
Next step
Put your own locations through the comparison.
Your first location report is free at any size. Bring a CSV of weekly sales and spend, or connect your POS and ad accounts.
Written by The Countlift team at First Byte in The Woodlands, Texas. Questions or corrections? Email hello@countlift.com.