Minimum detectable effect for marketing tests, explained

The minimum detectable effect is the smallest real lift your marketing test is likely to catch. If your ads truly add less than that, the test will probably say “can’t tell”, however good the ads are. You can calculate it before launch from data you already have. You should, because it decides whether the test is worth running at all.

Most marketing tests that come back “not significant” could never have come back as anything else. Advertising effects are small next to the everyday noise in sales. Across 25 large field experiments run with major US retailers and brokerages, most reaching millions of customers, the median confidence interval on return on investment was more than 100 percentage points wide (Source: Lewis and Rao, Quarterly Journal of Economics, 2015). Multi-location brands measure locations rather than individual shoppers, which helps a lot. The lesson still stands: size the test first.

What minimum detectable effect means, in plain English

Think of a test as trying to hear a voice in a noisy room. The voice is the lift your ads cause. The room noise is how much your test locations drift against their comparison in an ordinary week. The MDE is how loud the voice has to be for you to hear it eight times out of ten.

Two conventions define “hear it”. A 95% confidence level means you only declare a lift when a gap that size would turn up by chance less than 5% of the time if the ads did nothing. And 80% power means that if the real lift equals the MDE, the test detects it 80% of the time. The 80% default comes from the statistician Jacob Cohen, who argued that most researchers treat a false alarm as roughly four times worse than a miss (Source: Cohen, Statistical Power Analysis for the Behavioral Sciences, 1988).

How to calculate the minimum detectable effect for a marketing test

For a geo test read against a weighted twin of comparison locations, a planning version takes five steps:

1. Measure the noise. Over a pre-period of 26 weeks or more, compute the weekly percentage gap between your planned test locations and their twin. Take its standard deviation. Call it σ.
2. Scale it to the test length. Divide σ by the square root of the number of test weeks. That’s the standard error of the average gap.
3. Multiply by 2.8. That’s 1.96 for 95% confidence plus 0.84 for 80% power. The result is the MDE as a percentage lift.
4. Turn it into customers. MDE × weekly customers at the test locations.
5. Turn it into budget. Customers needed × a realistic cost per extra customer, from past reads or a sober guess. That’s the least you’d need to spend per week to expect a readable result.

The formula assumes weeks are independent of each other. They aren’t quite, so treat the result as a slightly optimistic floor. Better tools replace the textbook step with simulations on your own history: pretend each steady market got the change, and look at how big the fake “lifts” get.

A worked example: sizing a test for a dental group

This example is illustrative. A 28-practice dental group wants to know whether an extra $2,000 a week of paid search and social brings in new patients. It plans an eight-week test in 10 practices, with the other 18 as the comparison. The 10 test practices see 220 new patients a week between them, 22 each. In the pre-period their weekly gap against the twin had a standard deviation of 7%. That’s high, but small weekly counts are always noisy.

The test can reliably see a 6.9% lift. The budget should buy about 5.1%. Run as planned, the test will most likely come back inconclusive, and someone will conclude that search and social “don’t work” for new patients. That’s a $16,000 way to learn the wrong thing. (The $180 per extra new patient is an assumption; our guide to dental marketing ROI per practice covers how to estimate it.)

Three ways to fix it

Running longer works because noise falls with the square root of the weeks: 2.8 × 7% ÷ √16 = 4.9%. Adding practices works because, if each practice’s noise is roughly independent, doubling them cuts σ by √2, to about 4.95%. Both just clear the bar, and both cost $32,000.

Spending more per practice is the cheapest fix here, because it raises the signal instead of only lowering the noise: $3,000 ÷ $180 = 16.7 extra patients a week, or 7.6%. Two catches. Returns may flatten at the higher spend, so $3,000 might buy less than 1.5 times what $2,000 does. And with only 10 practices, one bad month at one practice still moves the total. Adding practices finishes in half the time of running longer, but it shrinks the comparison pool to eight, which is thin.

How to lower the minimum detectable effect

• Count customers, not revenue. Revenue adds ticket-size noise that has nothing to do with the ads. Transactions, guests, new patients or booked jobs are steadier.
• Use weekly totals. Daily data adds day-of-week swings without adding much information, unless your method handles them.
• Drop the wild locations. A location being remodelled, one that just opened, or one with a new manager adds noise to whichever group it’s in. New stores need their own read; see how to measure a grand opening.
• Pool locations into markets. A market of five locations is steadier than any one of them.
• Concentrate the spend. Put the test budget into fewer markets at a higher level rather than spreading it thin.
• Pick a clean pre-period. If the last 26 weeks included a menu overhaul or a price rise in half your markets, the noise estimate will be inflated.

Reading test results against the MDE

The MDE also tells you how to read whatever comes back. The 95% range around a result is roughly 0.7 times the MDE either side of the estimate, so a test with a small MDE gives a tight range and one with a big MDE gives a loose one.

• Range clear of zero. The test saw an effect. Report the range, not just the middle.
• Range spans zero but excludes the lift you expected. A fair test came back empty. The channel probably adds less than you hoped. That’s a real finding, and it should move budget.
• Range spans zero and the lift you expected. The test couldn’t tell. Say so plainly. It isn’t evidence against the channel.

One trap runs the other way. When an underpowered test does come back significant, the estimate is usually too high, because only the lucky draws clear the bar. If a test with a 7% MDE shows a 9% lift, don’t budget on 9%. Re-run it bigger, or plan on something closer to what you expected going in.

Common mistakes with minimum detectable effect

• Borrowing a website A/B test calculator. Those assume thousands of independent visitors. A geo test has a few dozen locations and a few weeks of data. The arithmetic is different.
• Assuming the lift you hope for. Base the expected lift on a cost per incremental customer you’d actually believe, not on the platform’s reported return.
• Sizing on one outcome and reading another. An MDE worked out on revenue says nothing about transactions. Pick the outcome first and size against it.
• Forgetting the ramp. If the first two weeks run at half strength, the average lift over a short test is lower than the steady lift. Our guide on how long a geo test should run shows the effect.
• Treating the MDE as a target. It’s a floor for what the test can see. Budget for an expected lift comfortably above it, say 20% higher, to leave room for the formula’s optimism.
• Starving the comparison pool. Moving locations into the test group makes the twin worse. Keep at least as many comparison locations as test ones; twice as many is better.

Questions people ask about minimum detectable effect

Work out the MDE before the media plan, not after the readout. It’s the cheapest five minutes in marketing measurement.

Written by The Countlift team at First Byte in The Woodlands, Texas. Questions or corrections? Email hello@countlift.com.