Designing a Geo Holdout Test: Market Selection, Duration and Statistical Power

A geo holdout test asks a media channel the most honest question available: what happens to sales in the places where we stop? No user-level tracking, no consent banner, no platform attribution model marking its own homework. You switch advertising off (or up) in some regions, leave it running in comparable ones, and measure the gap. Done well, it produces the causal number that attribution can only approximate, and the calibration point that keeps a media mix model anchored to reality.

Done badly, it produces a confident-looking number that is mostly noise, and in Australia it is unusually easy to do badly. We covered why experiments sit at the top of the evidence hierarchy in Incrementality Testing: The Gold Standard. This guide is the practical sequel, with a design-choices table, a worked power calculation you can reproduce in a spreadsheet, and the pre-launch checklist we use.

Start With the Decision, Not the Channel

Every good geo test begins with a sentence like: "If incremental return on ad spend (iROAS) is below X, we will cut the channel; if it is above Y, we will scale it." That sentence sets everything downstream: which KPI to measure, what effect size matters, and therefore how large and long the test must be.

The fault we find most often is the reverse order. Someone decides to "run a GeoLift on Meta", picks two states, runs for three weeks, and discovers the confidence interval spans everything from "disaster" to "best investment we have". An experiment that cannot change a decision is an expensive way to produce a slide.

Two other choices belong here. The KPI must be measurable by geography from a source the channel cannot influence: your order system or CRM by delivery postcode, not platform-reported conversions. And you must decide between a holdout (spend off in treatment regions) and a heavy-up (spend increased), because they answer different questions.

Choosing the Geographic Unit in Australia

In the United States, geo experiments are usually built on Nielsen's DMAs, roughly 200 self-contained media markets. Australia offers nothing so convenient. According to the ABS, Greater Sydney and Greater Melbourne were home to 5.64 million and 5.44 million people at 30 June 2025, together about 40% of the national population of 27.6 million (ABS Regional Population). New South Wales and Victoria together hold about 57%, and for most national advertisers those two cities drive an even larger share of revenue. That creates three problems.

1. Few units. A state-level design has eight units, two of which dominate. You cannot randomise meaningfully across eight units, and holding out NSW is a commercial non-starter for most businesses.
2. No twin for the giants. Nothing looks like Sydney except Melbourne. If either city is in treatment, the control group struggles to reproduce its behaviour.
3. Small markets are noisy. Hobart, Darwin and regional areas can be useful treatment cells, but their weekly sales are often small enough that random variation swamps any plausible media effect.

The practical answer is to go below state level:

• Metro and regional TV markets. The closest thing Australia has to DMAs, and essential when the test involves broadcast TV, radio or out-of-home bought by market.
• SA4 regions. The ABS designs Statistical Area Level 4 regions around labour markets, with city SA4s typically holding 300,000 to 500,000 people. They split Sydney and Melbourne into many units, are stable over time, and can be mapped to postcodes with ABS correspondence files (imperfectly, as some postcodes straddle boundaries).
• Postcode clusters. The most flexible option for digital channels, and the hardest to keep clean, because postcode boundaries and platform location signals rarely line up perfectly.

Whatever the unit, you must be able to both target it in the ad platforms and report on it in your sales data.

Matching Treatment and Control Markets

Randomisation works with hundreds of units. With the tens realistic in Australia, pure random assignment often produces groups that differ in size, seasonality or trend, and the imbalance shows up as a fake "effect". The documentation for Google's open-source matched markets library makes the same point: geo heterogeneity and small samples leave randomised groups unbalanced, which is why matched designs exist.

The standard approaches:

• Pre-period correlation matching. Pull at least a year of weekly KPI data by unit, preferably two, and search for the treatment and control combination whose aggregated series track each other most closely. You want high correlation and a stable ratio, not just similar averages.
• Stratified or matched pairs. Group units into strata (metro or regional, state, size), pair similar units and randomly assign one of each pair to treatment. This keeps the protection of randomisation while forcing balance. Google's Trimmed Match package is built around this paired design.
• Synthetic control. Build a weighted combination of untreated units that reproduces the treatment group's pre-period behaviour. This is how Meta's open-source GeoLift package works, and it suits Australia because it can construct a counterfactual for a cluster that no single market resembles.

Validate any design with placebo tests: pretend a test started eight weeks before the end of the pre-period and run the full analysis. A good design shows no "effect" where nothing happened. If your placebo shows a 6% lift, your real test cannot reliably detect anything smaller.

A rule we apply: keep Sydney and Melbourne in the control group unless the decision specifically requires testing there. They are expensive to hold out and hard to match. A treatment group of mid-size SA4s across several states usually tests the channel more cheaply and more precisely.

Holdout Versus Heavy-Up Designs

A holdout switches the channel off in treatment regions and measures what is lost. A heavy-up increases spend there, often doubling it, and measures what is gained.

The treatment shares are our working ranges, not rules. The distinction matters because returns diminish: a channel can have a healthy average return and a poor marginal return on the next dollar. If the decision is a budget increase, a holdout answers the wrong question, the same logic behind the response curves in media mix modelling.

For search, brand and non-brand behave very differently, and a geo holdout on brand search is one of the cleanest ways to settle the cannibalisation argument, as we discussed in paid search incrementality.

Choosing the Test Duration

Power rewards longer tests; cost and contamination punish them. Four constraints set the floor.

1. Cover the purchase cycle. The test must run long enough for a typical customer to go from exposure to purchase. For B2B leads or big-ticket purchases that can be months, which may push you to a leading indicator such as qualified enquiries.
2. Add a cool-down window. Media effects persist after spend changes, so keep measuring for at least one purchase cycle after the test ends.
3. Use whole weeks. Day-of-week patterns are strong in almost every business.
4. Avoid structural breaks. Black Friday (27 November in 2026), Boxing Day, end of financial year and Click Frenzy all distort sales unevenly across regions. So do state-specific events: school holidays that differ by state, the Melbourne Cup holiday in Victoria, football finals. A test starting this month will collide with peak season; plan for February or March instead.

In our experience, most digital holdouts for consumer brands need four to eight test weeks plus a two to four week cool-down. The worked example shows how to check your own numbers.

Statistical Power and Minimum Detectable Effect: A Worked Example

The minimum detectable effect (MDE) is the smallest true effect your test has an 80% chance of detecting at 5% significance (two-sided), the usual conventions. If the effect your decision depends on is smaller than the MDE, the test is underpowered, and you should change the design before launch, not after.

The Assumptions

• A retailer tests Meta prospecting with a holdout. The treatment group is a cluster of SA4s producing AUD 400,000 of revenue per week, about 20% of national sales.
• Meta spend in those regions is AUD 15,000 per week, which will be switched off.
• A model predicting treatment-group revenue from the control group has a weekly residual standard deviation of 5% of revenue, or AUD 20,000, estimated from placebo tests.
• The model is fitted on 52 weeks of pre-period data.
• Platform-reported ROAS is 4.0. Given its margins, the business's breakeven iROAS is 2.0.

The Maths

The test estimates average weekly lift: actual minus predicted revenue, averaged across the test weeks. With n test weeks and m pre-period weeks:

Standard error = residual SD x sqrt(1/n + 1/m)

MDE = (1.96 + 0.84) x standard error = 2.8 x standard error

1.96 = z value for 5% two-sided significance
0.84 = z value for 80% power

For an eight-week test: sqrt(1/8 + 1/52) = sqrt(0.1250 + 0.0192) = sqrt(0.1442) = 0.380. The standard error is 20,000 x 0.380 = AUD 7,600, and the MDE is 2.8 x 7,600 = AUD 21,280 per week, or 5.3% of treatment-group revenue. Dividing by weekly spend gives an MDE in iROAS terms of 21,280 / 15,000 = 1.42.

Reading the Result

If the true iROAS is at breakeven, 2.0, switching Meta off removes 2.0 x 15,000 = AUD 30,000 a week, a 7.5% drop. That exceeds the MDE at every duration, so even four weeks gives at least an 80% chance of detecting it, and eight weeks a comfortable margin. If the true iROAS is 1.0, the drop is AUD 15,000 (3.75%), below the MDE at every duration. A twelve-week test would probably come back "not significant", and that would not prove the channel does nothing.

This is the real value of a power calculation: it tells you which conclusions the test can support. Here, eight weeks can reliably show whether Meta clears breakeven, but cannot precisely measure a low iROAS. To distinguish 0.5 from 1.0 you need more power, and the levers, roughly in order of effectiveness, are a better-matched control (lower residual SD), a larger treatment share, a longer test, or a bigger spend change.

Two caveats. The formula ignores autocorrelation between weeks and uncertainty in the matching, both of which widen real intervals, so treat it as an optimistic floor. And the purpose-built tools do this properly: GeoLift and Google's packages run simulation-based power analysis across candidate market combinations, which is what we use for client designs.

Contamination: How Geo Tests Leak

A geo test assumes treatment regions get the treatment and control regions do not. Several things blur that line.

• Platform geo-targeting accuracy. Google Ads' default location option, "Presence or interest", can reach people outside a region who have shown interest in it. For tests use "Presence", and note Google's own help documentation calls location targeting best effort. Expect some leakage and keep buffer regions between treatment and control where you can.
• Commuter spillover. People live in one SA4, work in another and browse on mobile in both. Whole labour-market regions leak less than scattered postcodes.
• National media. TV, press, sponsorship and PR land everywhere. They are fine if constant; a national campaign launching mid-test is not.
• Exclusions and compensation. Every campaign on the tested platform, including automated ones such as Advantage+, needs the same regional exclusion, or the held-out regions quietly receive ads. Switching Meta off can also lift branded search or direct traffic, so measure the total business KPI and freeze other channels' budgets where you can.
• Supply-side differences. Stockouts, store openings, price changes or a regional competitor launch can masquerade as media effects. Agree a change freeze and log what you cannot freeze.

Analysing the Result

Choose the analysis method during design, not after seeing the data. The main options:

• Difference-in-differences. Compare the treatment group's change from pre-period to test period with the control group's change. Simple and transparent; its key assumption, parallel trends, is exactly what matching protects.
• Time-based regression (TBR). Regress treatment KPI on control KPI in the pre-period, then predict what treatment would have done without the change. This is the approach in Google's matched markets library, and suits designs with few units.
• Synthetic control. A weighted blend of control units forms the counterfactual. GeoLift implements this with built-in inference.
• Trimmed Match. For randomised paired designs, it estimates iROAS while trimming anomalous pairs, so one odd region cannot dominate.

These methods trace back to Google's 2011 paper by Vaver and Koehler, Measuring Ad Effectiveness Using Geo Experiments, still the clearest short explanation of why the design works. Both Google repositories state they are not officially supported Google products and are for research purposes; GeoLift sits in Meta's incubator organisation. All three need someone comfortable in R or Python.

Report the result as iROAS with a confidence interval, against the decision rule you wrote first. "iROAS 1.8, 95% interval 1.1 to 2.5, against a breakeven of 2.0" is a useful finding: the channel clearly drives sales, but may not be profitable at its current scale, which points to testing a lower spend level next. A single geo test is a calibration point, not a permanent truth: it should update your MMM and sit alongside attribution, the triangulation approach we recommend. For campaign-level questions inside Google Ads, such as bidding strategies, the built-in experiments tooling is usually the better instrument; geo tests are for channel and budget questions.

Pre-Launch Checklist

If any item is not ticked, delay the launch. A week's delay is cheaper than an unreadable result.

1. The decision rule, breakeven iROAS and action at each outcome are written down and signed off by the budget owner.
2. The KPI comes from a source the channel cannot influence, available weekly by geographic unit.
3. At least 52 weeks of weekly KPI history by unit is pulled and cleaned, with anomalies documented.
4. Units can be targeted in every affected platform and reported in the sales data, with the postcode mapping saved.
5. Treatment and control were selected by a documented method, and the selection is saved.
6. Placebo tests show no spurious effect larger than the effect you need to detect.
7. A power analysis shows the MDE, in iROAS terms, sits below the effect the decision depends on.
8. The test runs in whole weeks, avoids sales events and state holidays, and includes a cool-down.
9. Presence-only targeting and regional exclusions are applied to every campaign that could serve the treatment regions.
10. Budgets, bidding, pricing, promotions and national media are frozen or logged, with merchandising and sales briefed.
11. The analysis method and primary metric are pre-registered in writing, so nobody can choose a flattering cut later.
12. A named owner checks delivery daily in week one to confirm the treatment regions are actually dark (or actually heavier).

If you want help designing or analysing a geo test, from market selection and power analysis through to calibrating your media mix model with the result, it is core work for our incrementality testing practice. Get in touch.