Turn It Down: Cost by Conversation Length Simulator
AI is stateless, so API costs compound quickly. How much more do long conversations cost than short ones?
Monthly Cost per Conversation Length
users/moconversations/user50,000conv/mo
Simplified approximation — system prompt cached after turn 1, re-read cheaply on subsequent turns. New user/assistant tokens written to cache each turn. Anthropic cache writes carry a 1.25× premium; reads are 0.1×. Turn 1 costs slightly more with caching enabled.
Tokens per conversation
Cumulative Conversation Tokens combines input (the full history re-sent each turn) and output tokens — everything billed across the conversation. Beyond the third turn, later messages are sized by the Turns 4+ Input and Output selectors — set separately, since models run far wordier than users — and tokenized per model.
Example conversationEdit to explore cost scenarios
System prompteditable
User
We're considering deploying an LLM-powered customer service chatbot. What ROI metrics should we track?
Assistant
Focus on three categories: cost deflection, quality, and speed. On cost, track handle time reduction and the percentage of tickets fully resolved without human escalation—this is your clearest line to dollar savings. On quality, watch CSAT and first-contact resolution rates, since a chatbot that frustrates customers creates downstream churn costs that easily exceed the savings. On speed, measure median time-to-resolution; customers value quick answers more than human ones, and this is often where LLMs outperform scripted IVRs. Set a 90-day baseline before any LLM goes live.
User messageeditable
Assistant
Assume your current blended cost per ticket is $12, handling 10,000 tickets monthly—that's $1.44M annually. A well-tuned LLM chatbot typically deflects 40–60% of volume at roughly $0.08–0.20 per deflected ticket in API costs. At 50% deflection, that's 5,000 tickets at $1.00 each in API costs versus $60,000 in agent costs—a monthly saving of roughly $59,000, or $708K annually. Against a $200–400K implementation and year-one tuning cost, you're looking at a 12–18 month payback. The variable that kills these projections is containment rate: if your chatbot escalates 70% instead of 50%, the economics invert almost immediately.
User
What implementation costs are teams usually underestimating?
Assistant
Three costs routinely get sandbagged: data preparation, evaluation infrastructure, and organizational change. Data is the largest surprise—cleaning, labeling, and structuring historical tickets for retrieval or fine-tuning typically runs 2–3x what teams budget. Evaluation is next: without a rigorous way to measure response quality at scale, you won't catch regressions before customers do, and building that infrastructure costs real engineering time. The third is change management—contact center agents whose queue shrinks by half need new roles or they leave, and attrition costs rarely appear in AI project budgets. Together these add 60–80% to the headline implementation number.
Token counts are measured per model's own tokenizer — which is why lines differ between providers. The first three turns use this real conversation; later turns repeat the lengths chosen with the Turns 4+ Input and Output selectors, tokenized per model. In a real multi-turn thread, every reply re-sends the full prior history as input — that's what drives the quadratic cost curve.
✦✦✦
K
Kerstin
AI assistant
I'd love to keep talking — enter your email and I'll send you a link to continue.
Want me to send this conversation to Kerstin so she can pick it up with you directly?
Powered by AI · may make mistakes · not professional advice