MMM vs Incrementality Testing: Which Question Should Each Answer?

MMM vs incrementality testing is usually framed as a choice. Should you build a model or run an experiment?

That is not how we use them.

We often start with a first-pass MMM using uninformed priors. At this stage, we want to see what the historical data can identify before we tell the model too much about what we think the answer should be. Some channels will have a clear signal. Others will have very little variation, move with other channels or produce a wide range of plausible contributions.

That first model is a diagnostic. We do not use it as the final answer on where the budget should go. It tells us where the data looks strong, where it looks weak and which parts of the story need more evidence.

We then go into the observed layers of the business. We look at market behaviour, buyer evidence, buyer behaviour and what actually happened inside the channels. Where we still need a causal answer, we create stronger evidence through incrementality testing. We then bring that evidence back into the MMM so that the final model has less freedom to fill gaps with assumptions. This is consistent with the broader Growth Dynamics evidence philosophy: each method sees part of the system, and the method should follow the decision.

The workflow looks like this:

First-pass MMM → observed evidence → incrementality where needed → final MMM

The first MMM finds the questions. The work in the middle gives us better evidence. The final MMM puts the portfolio back together.

The first MMM shows us where the signal is

Historical data does not contain the same amount of information about every channel.

A large channel that has changed spend many times may give the model useful variation. If those changes are followed by clear movements in revenue, the signal can be strong enough for the data to constrain the contribution quite tightly.

Other channels are much harder. A channel may barely change spend for two years. Two channels may always move together. A smaller channel may create a real effect that is tiny compared with normal changes in weekly revenue. Model sophistication cannot create information that was never present in the data.

The first MMM helps make those differences visible. We can see where the model appears to be learning from clear historical variation and where the contribution depends much more on the structure around the data.

That gives us a better research plan. A weak or strange contribution does not immediately become a budget cut. It becomes something we need to understand.

Then we look at what actually happened

A model can tell us that a channel appears to have contributed a certain amount. Before taking that number too seriously, I want to understand what happened in the business during the period the model is trying to explain.

The first four layers of our framework are observed. We look at whether branded and competitive demand moved, what buyers say drove consideration and choice, where meaningful intent appeared and where the money actually sat inside the channels. These layers cannot prove causality, but they can answer many questions without asking a statistical model to infer them.

This becomes especially important when the MMM compresses a complicated history into one channel coefficient.

Imagine two years of Meta spend. During that period the company may have changed its creative many times. Campaign objectives may have changed. Audiences, placements, spend levels and bidding may all have changed too.

Calling all of that "Meta" does not make it one stable treatment.

Our framework treats creative as part of the treatment for exactly this reason. Channel effectiveness changes when the creative portfolio changes, and a coefficient averaged over years may describe a portfolio of activity that no longer exists.

There is good external evidence for taking creative seriously. Thinkbox describes creativity as the most powerful driver of advertising effectiveness within marketers' control, while System1 describes creativity as the strongest driver of advertising profitability that marketers control.

That creates a problem for any claim about the permanent "performance of a channel."

If the creative changes, the treatment changes.

Incrementality gives us a cleaner answer to a narrower question

When the observed evidence is not enough and the budget decision needs causality, we move to incrementality.

Incrementality testing creates variation on purpose. Instead of asking historical data to separate everything that happened at once, we change a defined treatment and estimate what changed because of that intervention.

That gives us stronger causal evidence for the question being tested.

The boundary of the result matters. An incrementality test on Meta measures the Meta activity that actually ran during the test. It had a certain level of spend, a certain creative portfolio, a certain bidding system, a certain audience and a certain set of market conditions. Our framework therefore treats the bidder and creative as part of the treatment, and it treats a local test result as evidence about the tested conditions rather than an automatic national truth.

If that test produces a strong lift, we have learned something valuable. We have not discovered a permanent property of Meta.

Change the creative and the response can change. Increase spend enough and marginal returns can change. Test different markets and the effect can change. Let the platform deliver to a different audience mix and the treatment can change again.

This is why I dislike statements such as "we proved Meta is incremental."

We proved that a defined Meta treatment created a certain effect under the conditions we tested.

That is a much more useful statement because it tells us what evidence we actually have.

MMM has the same treatment problem over a much longer period

Incrementality has a clear treatment window. MMM often has the opposite problem: it looks across years of changing activity and tries to estimate a useful channel-level relationship.

A two-year Meta coefficient could average together excellent creative and terrible creative. It could contain prospecting campaigns, retargeting campaigns, different optimisation goals and major changes in spend. The platform itself may also change during the period.

The model still has to produce a usable representation of that history.

The resulting coefficient therefore belongs to the historical portfolio of Meta activity that existed in the data. Treating it as the expected return from the next Meta dollar assumes that the future treatment will behave enough like the historical one.

Creative makes that assumption especially uncomfortable because it can materially change advertising effectiveness. A channel can look much better next year with stronger creative, or much worse with weaker work, even if the media platform has not changed.

Incrementality narrows the treatment enough to improve causal identification. MMM widens the treatment enough to help with portfolio allocation.

Both lose information when we turn the result into a statement about the channel itself.

Both methods eventually hit signal-to-noise

There is another limit that neither method escapes.

Signal-to-noise.

Incrementality has an advantage because the experiment deliberately creates variation. A large treatment that produces a clear response can stand out from normal business movement much more easily than a relationship reconstructed from messy historical data.

The problem returns when the effect is small or slow.

Imagine a campaign creates a meaningful amount of incremental revenue within a week. That effect has a better chance of standing out from normal revenue variation.

Now imagine the same total effect is spread across six months. During those six months prices change, competitors move, promotions happen, distribution changes, creative changes and the normal volatility of the business continues. The original marketing effect becomes much harder to separate from everything around it. This signal-to-noise problem with long-term effects is already a core part of the Growth Dynamics evidence philosophy.

MMM can give an effect a long adstock and allow marketing to influence revenue months later. That gives the model a way to represent a long-term effect. It does not make the long-term signal in the historical data stronger.

Incrementality runs into the same problem in a different way.

Extending the test window can be useful when customers need more time to convert. It does not guarantee that a small effect spread over a long period will become detectable. A longer test also creates more time for other parts of the business to move.

Continuing to observe the post-treatment period can also help, but the same information problem remains. Once we start estimating how long the effect persists after the treatment ends, assumptions about the decay of that effect start to matter.

A longer test window cannot manufacture signal. A longer adstock cannot manufacture signal either.

Long-term effects are difficult because the causal signal gets spread across more time and more noise. Our belief map states this directly: long consideration journeys dilute causal signal, and a longer test window does not automatically recover the impact.

Incrementality should feed the MMM

After the first MMM and the work that follows it, we know much more about the system.

We may know that a channel the first model struggled with produced a clear effect when spend was changed deliberately. We may know that a response curve suggested by the model conflicts with what happened when the business actually moved the budget. We may know that one period of a channel contained much stronger creative than another.

That information should not live in a separate measurement report.

It should come back into the model.

Growth Dynamics puts MMM at the modelled end of the evidence ladder because it can consume what we learned below it. Surveys can inform priors. Incrementality can provide causal anchors. Mid-funnel evidence can help us understand timing. Creative differences can tell us when a stable channel coefficient is a poor description of what actually happened.

The assumptions in the final MMM now have evidence behind them.

That does not make every parameter objectively true. It means we are no longer asking the model to settle every causal question from correlated spend and revenue alone.

There is something outside the model capable of challenging what it says.

The final MMM solves a problem the individual test cannot

Incrementality is strongest when the question is specific.

What happened when we changed this investment? How much lift did this treatment create? Did revenue fall when we removed this spend? Is the current level of investment still incremental?

A CMO eventually has to solve a wider problem.

There may be ten channels competing for the same budget. Increasing one means taking money from somewhere else. Marginal returns matter across the whole mix, and it is rarely practical to run an experiment for every channel at every possible spend level.

The final MMM helps connect that portfolio.

By this stage, we have a first read of where the historical signal is strong and weak. We have checked the story against observable evidence. We have created causal evidence for the questions that justified a test.

The model now has a much better foundation for the strategic allocation problem.

That is the role I want MMM to play.

So, MMM or incrementality?

For us, the answer is rarely a clean choice between the two.

We use a first-pass MMM to see what the client's historical data can identify. We then investigate the important parts of the model with observed evidence. When the allocation question still needs causality, we use incrementality to create stronger evidence. The final MMM brings that evidence back together and helps us allocate across the portfolio.

Each method keeps its proper boundary.

Observed evidence tells us what happened in the market, with buyers and inside the channels. Incrementality estimates what changed because of a defined intervention under specific conditions. MMM estimates the wider historical mix and gives us a framework for thinking about allocation across it.

Creative means the treatment is always more specific than the channel name suggests. Signal-to-noise means neither experiments nor MMM can recover effects that the data does not contain clearly enough. Long-term effects make that problem worse because the signal is spread over more time while the rest of the business keeps changing.

The first MMM tells us where the data has something to say.

The observed and experimental work gives us stronger evidence where it does not.

The final MMM uses that evidence to help decide where the next dollar should go.