Geo holdout tests for multi-location brands: a practical guide

Observational comparisons — markets where spend changed versus markets where it didn’t — are the everyday workhorse of measurement for multi-location brands. But sometimes you need a cleaner answer: whether a new channel works at all, whether a channel you have run for years still earns its budget, or how much a market responds to doubling spend. For those questions, plan a geo holdout test.

A geo test is a controlled experiment in which markets, not people, are the unit. You change spend in some markets, keep it as it was in others, and compare outcomes at your locations. It needs no cookies or user-level tracking, which makes it one of the few methods that has become more useful, not less, as tracking has declined.

What a geo holdout test is

There are two common designs:

• Go-dark (holdout). A channel keeps running in most markets and is paused in a selected set. If outcomes in the paused markets fall relative to their comparison, the channel was doing something.
• Scale-up (heavy-up). A channel is added or increased in a selected set of markets while the rest continue as before. This answers “what would more spend buy?” and is often an easier conversation internally than switching something off.

In both cases the logic is that of any experiment: the only planned difference between test and comparison markets is the media change, so a difference in outcomes can be attributed to it.

Step 1: Write down the question and the outcome

A good test answers one question, measured on one primary outcome. “Does paid social drive new member sign-ups at our clubs?” is testable. “Does our marketing work?” is not. Choose the outcome before the test starts — weekly guests, new patients, booked jobs — and write down what result would change your plans. If no plausible result would change a decision, don’t run the test.

Step 2: Choose test and holdout markets

Use the geography your media is actually bought in. In the US that is usually the Nielsen Designated Market Area (DMA); Nielsen divides the country into 210 of them. Smaller units such as ZIP codes can work for channels that target precisely, like search and some social, but they spill over more easily.

Then pick markets carefully:

• Match on history. Test and comparison markets should have tracked each other closely over the past several months to a year. Similar trends matter more than similar size.
• Randomize within pairs where you can. Group markets into similar pairs or small groups, then randomly choose which one gets the change. Hand-picked test markets tend to be the ones you are most optimistic about.
• Exclude the messy ones. Leave out markets with openings, closures, remodels, major promotions or known disruptions during the test window.
• Mind spillover. Neighboring markets share commuters and media signals; a campaign targeting one market reaches people in the next. Avoid placing test and comparison markets side by side, or leave a buffer market out of both groups.
• Keep enough comparison markets. A single holdout market makes for a fragile read. Several, combined into a weighted twin, are far more stable.

Step 3: Decide how long to run it

Length is driven by three things: how often customers buy, how long ads take to work, and how noisy weekly outcomes are.

• Cover several complete purchase cycles. A coffee chain sees regulars every few days; a dental group may see a new patient book weeks after first seeing an ad.
• Use whole weeks, so every test week contains the same mix of weekdays and weekends.
• Avoid windows that straddle major holidays unless the comparison markets experience them in the same way.
• Plan a post-period of a few weeks after the media change ends. Effects often linger, and in go-dark tests the decline can lag.
• Have a clean pre-period at least as long as the test, to fit the comparison and estimate the noise.

Step 4: Size the test — budget and minimum detectable effect

The minimum detectable effect (MDE) is the smallest lift a test has a good chance of detecting, given the noise in your data. Size it before launch. A test that can only detect a 10% lift is useless if the best realistic outcome is 3%, and running it will most likely produce a “no significant effect” that tells you nothing.

A simple approximation is enough for planning. Measure how much the weekly gap between the test markets and their comparison wanders during the pre-period — its standard deviation, in percent. For a two-sided test at the conventional 5% significance level with 80% power, the MDE is roughly 2.8 times the standard error of the average gap over the test. (2.8 is the sum of the matching normal quantiles, 1.96 and 0.84.)

A worked example (illustrative)

Say the weekly gap between your test markets and their twin has a standard deviation of 3% in the pre-period. Over an eight-week test, if weeks were independent, the standard error of the average gap is about 3% ÷ √8 ≈ 1.1%, and the MDE is about 2.8 × 1.1% ≈ 3%. Weekly outcomes are usually correlated from one week to the next, which reduces the effective number of independent weeks, so treat this figure as optimistic.

Now translate it into budget. If the test markets serve 20,000 customers a week, a 3% lift is 600 customers a week. If your best guess for the channel’s cost per incremental customer is $25, you would need around $15,000 a week of incremental spend to expect an effect large enough to detect. Too little budget is the most common reason geo tests come back inconclusive.

Step 5: Read the result honestly

When the test ends, compare each test market with a weighted blend of comparison markets fitted on the pre-period — the synthetic control approach — and report the lift with an interval, not just a point estimate. Two checks make the read more trustworthy:

• Placebo tests. Run the same analysis pretending each comparison market was the test market. If many of them show “lifts” as large as your real one, your result is within the noise.
• Pre-period fit. If the blend didn’t track the test market well before the test, the comparison after it isn’t reliable either.

Report the result in terms the business can act on: incremental customers, cost per incremental customer, and the range. A result whose interval spans zero is not proof of no effect. It means the test couldn’t tell, and you should say how large an effect it could have missed.

Common mistakes

• Underpowered tests. Running a test that can’t detect any realistic effect, then reading “not significant” as “doesn’t work.”
• Changing other things mid-test. A price change, a new promotion or a shift in another channel in only some markets contaminates the comparison.
• Peeking and stopping early. Checking every week and stopping the moment the result looks good inflates false positives. Fix the length up front.
• Convenience market selection. Testing in the markets a regional manager volunteered, or the ones with the best stores.
• Ignoring spillover. Adjacent test and comparison markets that share media coverage or customers understate the true effect.
• Counting new stores as lift. Locations that opened shortly before or during the test inflate whichever group they are in.
• Measuring clicks instead of customers. The point of a geo test is to measure outcomes at your locations. Using platform conversions as the outcome reintroduces the problem the test was meant to solve.
• Testing once and never again. A channel’s effect changes with creative, competition and saturation. Re-test the channels that carry the most budget periodically.

A well-planned geo test takes a few hours of setup and a couple of months of patience. In exchange you get an answer about your own customers at your own locations that no attribution report can give you.

Written by The Countlift team at First Byte in The Woodlands, Texas. Questions or corrections? Email hello@countlift.com.