What Does It Actually Cost to Run AI in Production?

Every provider publishes a price per million tokens, and every team that has run AI in production knows that number is the smallest line in the bill. The closing panel at AI Infra Signal — DCVC, Tensormesh, Together AI, and Nutanix, moderated by Stanford — was framed around power, cluster reliability, inference economics, security, and where value accumulates in the stack. Those are the inputs that actually decide cost. Here is how they add up.

Why is the token price a poor estimate of cost?

Because it prices one input and hides five. The same workload also pays for the share of your GPU fleet that sits idle, the data path feeding it, the power and reliability of the cluster, the retries and monitoring around each call, and the human review that catches what the model gets wrong.

How does GPU utilization change the unit cost?

Directly and brutally. A fleet running at half utilization has doubled the cost of every token it serves. Scheduling, workload placement, isolation between tenants, and reliability decide how much of the hardware you already pay for turns into output — which makes utilization a pricing decision rather than an ops detail.

This is the economics that neoclouds — cloud providers built around GPU capacity and AI workloads — compete on. Buying capacity is the easy half; keeping it busy with useful work is the half that sets the margin.

Where does the data path enter the bill?

A GPU waiting for data still costs what a GPU costs. Storage throughput, memory capacity, and network behaviour have to be evaluated alongside scheduling, because improving any one of them usually just moves the bottleneck rather than removing it.

The effect gets worse as context length and concurrency increase, which is exactly the direction most AI products move as they mature. A cost model built on a short-context prototype will understate a long-context production workload.

What do power and cluster reliability actually cost?

Power is a hard constraint on how much compute can be placed at all, and reliability is what decides whether a run finishes. A job that fails at 90% and restarts has paid for 190% of itself, so failure rates translate into cost as directly as hardware prices do.

How should an enterprise measure AI ROI?

At the workflow level, not the API level. The useful questions are how much time the workflow saves, whether quality improves, how often a person has to intervene, and what it costs to maintain. Inference spend is one input; integration work, retries, monitoring, and human review belong in the same calculation.

This is the gap that kills otherwise successful pilots. A workflow can be technically impressive and still fail to justify the resources it takes to operate, because the demo measured output quality and the business measures total cost of a finished task.

• Time saved per completed task, not per model call.
• Quality change against the process being replaced, measured the same way both times.
• Intervention rate — how often a human has to step in, and how long that takes.
• Maintenance cost: integration work, monitoring, evaluation, and the retries nobody budgets for.

Does model routing reduce cost?

It can, but it moves the decision rather than removing it. Once an application uses different models for different tasks, each task is a trade between quality, latency, cost, availability, and data requirements. A shared interface makes access simple; it does not tell you which model fits a workload.

A cluster at half utilization has doubled the price of everything it serves.

Where value accumulates in the stack

In the layers that make intelligence usable rather than the layer that produces it: supplying the right inputs, supporting execution, measuring results, and keeping the economics workable. That is the reason inference economics, data infrastructure, and enterprise ROI kept turning out to be the same conversation.