What Leaders Actually Need to Know About AI Right Now
The boardroom conversation about AI has outpaced most leaders' ability to evaluate what they're hearing. Here's how to ask better questions — and what the answers should sound like.
The most dangerous position a leader can be in right now is confidently wrong about AI.
Confidently right is ideal. Honestly uncertain is workable. But confident wrongness — the kind that comes from absorbing the narratives of AI vendors, breathless press coverage, and enthusiastic technical teams without the framework to evaluate them — leads to expensive mistakes.
This is not an indictment of leaders. The pace of change in AI has been genuinely unprecedented. The ecosystem of people who stand to benefit from your confusion and urgency is enormous. And the vocabulary of the field shifts fast enough that last year’s framework is already partially obsolete.
Here’s what I tell the C-suite leaders I work with.
The question is never “should we do AI”
That question was settled. The question now is: which AI initiatives, in what sequence, to solve which specific problems, measured against which outcomes.
Every AI initiative your organization considers should be able to answer four questions before it receives significant resources:
- What specific decision will this AI improve or automate?
- Who currently makes that decision, and what changes if a model makes it instead?
- What does success look like in 90 days? In 12 months?
- What’s the minimum viable version — and what would we learn from it?
If your teams can’t answer these cleanly, the initiative isn’t ready to fund at scale.
The ROI conversation is usually backwards
Most organizations evaluate AI ROI at the end of a project. By then, it’s too late — you’ve already spent the budget, and the measurement framework is reverse-engineered to justify a decision already made.
ROI thinking belongs at the beginning, not the end. Before any significant AI investment, you need a specific, falsifiable hypothesis about the business outcome. “Improve customer experience” is not a hypothesis. “Reduce churn by 8% in the enterprise segment within 18 months” is.
Data quality is usually the real constraint
The most common reason AI initiatives underperform is not model complexity, compute costs, or talent gaps. It’s data.
Specifically: data that is technically available but structurally unusable. Inconsistent definitions across systems, poor labeling practices, historical data that doesn’t reflect current business conditions, or data that was collected for reporting purposes and was never designed to train a model.
Before you invest in AI capabilities, invest in an honest audit of your data infrastructure. The ROI on data quality improvements often exceeds the ROI on the AI applications that depend on them.
What good AI leadership looks like
The leaders who get this right share a few habits:
They ask for second opinions. They build in explicit checkpoints where a trusted outside perspective — someone who isn’t selling them anything — can challenge the initiative’s assumptions.
They separate the vendor conversation from the strategy conversation. They form their own view of what they need before they evaluate solutions.
They reward honest assessments of failure. Teams that hide early signals of underperformance to protect their projects create the most expensive surprises later.
They distinguish between AI that automates existing decisions and AI that enables new ones. Both are valid. Neither is obviously superior. The choice should be deliberate.
Kerstin Frailey is a Cornell PhD, Johns Hopkins faculty member, and forthcoming author of a practical guide to AI and ML for business leaders. She has spent fifteen years leading data science teams across industries, from startups to post-IPO companies.