What it really costs to run LLMs in production

When teams budget for an AI feature, they price the model. Tokens in, tokens out, a number per thousand, and the spreadsheet looks reassuring. Then the thing ships, and the real costs arrive: the ones that were never on the token calculator. Running a language model in production is not the same as calling one in a prototype, and the gap between the two is where budgets break. It is worth knowing what actually sits in that gap before you commit to the roadmap.

The visible cost is the small one

Per-token pricing is genuinely the easy part to reason about, and for many products it is not even the largest line. It is visible, it scales predictably, and it gets cheaper over time. If the API bill were the whole story, most AI features would be trivial to run. It is not the whole story, and treating it as such is why so many pilots look cheap and so many production systems do not.

The costs that actually add up

The weight is in everything around the model. Retrieval infrastructure to feed it your data, and the work of keeping that data fresh and indexed. Evaluation and monitoring, because a model that silently degrades is worse than one that fails loudly, and you only know which you have if you are watching. Guardrails and the second model that checks the first. Human review of the cases the system is unsure about, which is a staffing cost, not a software one. Prompt and version maintenance as models change under you. Latency engineering when "correct in ten seconds" is not good enough. None of these show up in a prototype, and all of them show up in production. This is the unglamorous reality behind MLOps.

The cost that dwarfs the rest

There is one cost that can exceed all the others combined: a wrong output that reaches a customer or a decision. A hallucinated figure in a report, a bad answer to a client, an automated action taken on a false premise. The price of that is not measured in tokens or compute; it is measured in trust, rework, and sometimes liability. A system designed to be cheap per call but careless about being wrong is not cheap, it has just moved its largest cost off the spreadsheet and onto the business.

Why prototypes lie about cost

A prototype runs on clean inputs, gets judged by the person who built it, and never has to survive a bad day. Production runs on whatever arrives, gets judged by customers, and has to handle the inputs nobody anticipated. That is the same reason most AI pilots fail to reach production: the cheap, clean version was never the hard version. Budgeting from the prototype is budgeting for a system you are not going to run.

How do you budget for this before you build?

A workable method is to budget in three layers rather than one line. The first layer is the model itself: tokens, or compute if you self-host, estimated from realistic volumes rather than demo volumes. The second layer is the machinery around it: retrieval and its data pipeline, evaluation runs, monitoring, guardrails, and the engineering time to maintain prompts and adapt when a model version changes underneath you. The third layer is people: who reviews the low-confidence cases, how many per day at your volume, and what that costs in staff time. If you cannot fill in the third layer, that is the budget telling you the design is not finished, because every production LLM system has an exception path and someone has to staff it.

An illustrative example shows why the layers matter. A team plans a document-summarisation feature and prices it at a few cents per document from the token calculator. At production volume, the retrieval index that keeps summaries grounded, the weekly evaluation runs that catch drift, and the analyst hour a day spent reviewing flagged outputs each cost more than the API bill. None of those lines is a failure. They are what "working" costs, and the teams that see them in the budget phase ship features that survive, while the teams that discover them after launch tend to quietly shelve the feature instead.

What good budgeting looks like

Price the outcome, not the call. The number that matters is cost per correct, trusted result, and it includes retrieval, evaluation, guardrails, human review, and the maintenance that keeps the whole thing honest. A team that budgets this way ships AI that survives contact with reality. A team that budgets for tokens ships a demo and inherits a surprise. If you are planning an AI feature and want a realistic picture of what it costs to run rather than to demo, tell us what you are building, and see how we approach it under custom software.