Finance Already Ran This Experiment

Before the Gaussian copula blew up the financial system, Darrell Duffie kept getting invited to explain it. Banks would call the Stanford finance professor in, sit him down, and ask him to walk through the formula that was pricing the mortgage securities they were selling by the trillion. And every time, he told them the same thing: it isn't suitable for risk management or valuation. Paul Wilmott had said it earlier still, before the formula was even published, that correlations between financial quantities are notoriously unstable and no serious theory should be built on them. The warnings were on the record the whole way through.

They lost anyway, and the reason they lost is the part worth sitting with. The model wasn't ignored because nobody understood it. It was ignored because it fit the market exactly and it was making everyone rich. Once calibrated, it could reproduce market prices to the decimal. That precision felt like proof. Nassim Taleb put the actual problem in one sentence: people got excited about the copula because of its mathematical elegance, but the thing never worked. Elegance and a clean in-sample fit beat a structural objection, right up until the objection arrived as a phone call in 2008.

I spent enough time around trading desks to know this isn't a story about one bad formula. It's a story about a specific failure mode, and finance learned to fear it the hard way: a model that explains the past immaculately and predicts the future not at all. On a desk you find out fast, because the market bills you the same week. A backtest that looks brilliant and dies live costs you money you can count, so the culture hardened around a rule that sounds obvious and almost nobody follows. Fitting the history is not evidence. Surviving out of sample is.

Marketing measurement is running the same play, and it hasn't had its 2008 yet. Marketers are no less capable. The difference is the clock. When a marketing mix model overstates some channels, no P&L blows up on Monday. Budget quietly flows to the wrong place for a year, results drift, and by the time anyone notices, a dozen other things have changed and the model is never the suspect. The feedback loop that taught finance humility is missing, so the field keeps reaching for sophistication where it should be reaching for doubt.

Here is the specific thing being papered over. Long-term brand effects live below the noise floor. The signal is small, slow, and buried in aggregate data that moves for a hundred reasons, which is why even large randomized experiments often show low statistical power for advertising lift. The standard reference on the method says it plainly: small advertising effects are hard to identify in noisy data, and observational models without experimental grounding can substantially overestimate impact. The honest reading is that the data frequently does not contain the answer people are extracting from it.

What the industry does instead is invent methods that manufacture the answer and call it measurement. Take long adstock, the workhorse. Adstock is a decay curve you impose on the media input before you fit anything. Choose a long decay and you have told the model, by hand, that spend keeps working for months. The model then dutifully finds a long-term effect, which it could hardly fail to find, because you put it there. This is not a hypothetical. In a 2025 paper in Applied Marketing Analytics, Peter Cain, an econometrician who took a PhD in monetary economics before moving into marketing, ran the standard dual-adstock model on real consumer-goods and white-goods data and got exactly the significant long-term coefficients the method is supposed to produce. Then he refit the same data with a model that separates the underlying sales trend properly, and the long-term effect vanished. The high-retention adstock term dropped out entirely, revealing the original result as spurious correlation, with no persistent brand-building effect observable in either dataset. He is not gentle about the general case: he cites separate simulation work by Franses finding that even at a modest retention rate of 0.9, the adstock structure produces a spurious relationship in half of all simulated cases. The same paper walks through the other standard fixes and reaches the same verdict each time. Scaling factors simply assume the long-term effect exists, bearing no relation to the data. Bolting brand-equity metrics onto the sales equation just relabels indirect effects as long-term ones. Three techniques, one move: assume the conclusion, then let the machinery hand it back to you wearing a coefficient.

The reflex when you press on this is to say the assumptions are transparent, everything is documented, so there is nothing to hide. But disclosing an assumption is not the same as reducing your reliance on it. Writing "we assume a long adstock" in a methodology note does not make the long-term effect real. It means that when the number turns out to be an artifact of that choice, they can point to the note. Honest disclosure of a load-bearing assumption comes with an instruction to trust the output less. This kind comes with the output at full confidence and the caveat filed where no one deciding the budget will read it.

The other reflex is to say that all models are wrong and some are useful, which is true, and which now gets used to mean almost nothing. George Box said it to argue for stating your error bars, not for waving them away. There is a difference between a model that is wrong in the ordinary bounded way, off by a knowable amount for knowable reasons, and a model that is wrong because it is handing you your own assumption back as a result. The phrase collapses that difference. Point at a specific identification failure and the reply comes back that all models are wrong, as if that settled it, when what it does is decline to say which kind of wrong this one is.

None of this means the tools are worthless, and this is the line I would not let anyone blur. Marketing mix modeling is a real improvement on the attribution theater it replaced, and short-term promotional response is something you can genuinely establish. What it reads well is the big and the fast: large channels moving sales in the near term, where the signal is strong enough to stand out. It gets sold as a complete measurement solution, but it is one instrument that reads part of the picture. Small channels and long-term effects fail for the same reason, not two: both are small movements lost in a noisy series, so the model cannot resolve them and you have no choice but to measure them some other way. The problem is selling a partial model as the whole. The method did not measure a long-term effect, it assumed one. Whether the effect is truly there is a separate question the data was never able to answer. The failure is specific: it happens when a model built to read weeks is asked to certify effects that play out over years, and the field responds to that gap by dressing assumptions as findings rather than admitting the data ran out. Finance made exactly that error, at scale, with the smartest people in the room and real money on the line, and it took a global crisis to teach the lesson. The lesson was never that models are useless. It was that a model you have fallen in love with will tell you what you want to hear, and the more elegant it is, the longer you will believe it.

So the test is simple, and it is the same test a desk applies before it funds a strategy. Not does the model fit. Not is it sophisticated. Does the effect survive contact with data the model has not already been told to find. If it does, you have a measurement. If it does not, a cleverer method will not save you. There is nothing there to measure. You have a prior, and you have dressed it as a finding.