The Impact Hypothesis: The Missing Link in AI Projects
Data science teams spend months building models that technically work — and fail to move the business. The culprit is almost always the same: an unstated assumption between output and outcome.
Successful AI projects fail all the time. Not because the models underperform — but because the path from model output to business outcome was never explicitly defined.
There’s a step that most teams skip. Call it the impact hypothesis: the unstated assumption about how a prediction, classification, or recommendation will actually change something in the organization.
The Gap Nobody Talks About
A model that predicts customer churn with 89% accuracy is technically impressive. But if no one in the organization has a plan for what to do with those predictions — which customers to target, with what offer, through which channel, at whose direction — the model generates reports that get filed and forgotten.
The gap between “our model works” and “our AI initiative delivered ROI” is almost always here.
Four Reasons It Gets Skipped
1. Resource allocation bias. Data science is expensive, complex, and visible. Implementation is operational and invisible. Teams optimize for what they can measure and control — and the model is easier to measure than the change management required to act on it.
2. Implicit assumptions. When the impact hypothesis goes unstated, it becomes ownerless. No one is accountable for it, and no one can challenge it. It lives as a shared assumption that everyone believes and no one has examined.
3. Apparent simplicity. “Of course we’ll act on the predictions” seems obvious until you try to specify how. Simple-sounding hypotheses often contain enormous execution complexity that only surfaces when someone tries to design the actual intervention.
4. No language for it. Without a name, the concept is invisible. Teams can’t improve what they can’t identify. Naming the impact hypothesis makes it possible to scrutinize it.
What This Looks Like in Practice
Before any AI initiative goes to build, I ask three questions:
What does the model output? (A score, a label, a ranking, a prediction — be specific.)
What human or system action does that output trigger? (Not “the team will use it” — what, exactly, will they do differently?)
What business metric moves as a result, and by how much? (If you can’t specify this, you don’t have a hypothesis — you have a hope.)
These questions feel obvious. They almost never have good answers the first time you ask them.
The Payoff
Teams that explicitly state and examine their impact hypothesis before building ship AI that actually changes things. Not because their models are better — but because they designed the whole system, not just the model.
The impact hypothesis is the difference between an AI project and an AI initiative.
This essay is adapted from a talk delivered at Data Leaders conferences and an article originally published on the Metis Blog.