AI inference costs: compare the cost of a useful answer

A price per million tokens is a useful input to an AI budget. It is a poor substitute for the budget itself.

Two models can charge different rates, use different amounts of context and produce different numbers of acceptable answers. A cheaper request that needs repeated attempts or extensive correction may cost more to turn into completed work.

The comparison should start with a unit the business recognises: an accepted document extraction, a resolved request or a usable summary. Token consumption then helps explain the cost of producing that unit.

Infrastructure and FinOps leads also face a choice about who runs inference. In its May 2026 announcement, IBM presented Red Hat AI Inference on IBM Cloud as a managed service intended to remove GPU and runtime management from customers. IBM Cloud CTO Jason McGee framed the offering around moving AI from pilots into production. Those are supplier claims about the operating model, not evidence that a particular workload will be cheaper.

For a buyer, the comparison must therefore include the engineering work retained in each option. A lower model charge can coexist with higher platform support costs. The calculation below isolates request and review costs first, so the other costs can be added visibly.

Build the request cost first

For a simple text workload, multiply input tokens by the input rate and output tokens by the output rate, using the supplier’s billing units. Add separately billed services where they apply. Caching, batch processing, tool use and other pricing features must follow the chosen provider’s definitions; they cannot be assumed to work identically across models.

The following example uses invented prices and volumes to show the arithmetic. It is not a comparison of commercial providers. Both options process the same set of 1,000 tasks, with 2,000 input tokens and 500 output tokens per attempt. The attempt count includes retries, and the token volumes are assumed constant for simplicity.

Input or result	Model A	Model B
Input price per million tokens	£1	£2
Output price per million tokens	£4	£8
Cost per attempt	£0.004	£0.008
Total attempts	1,200	1,050
Total model charge	£4.80	£8.40
Accepted tasks	800	950
Model charge per accepted task	£0.006	About £0.0088

Model A remains cheaper on model charges per accepted task, despite more retries and fewer accepted results. The calculation does not force the more expensive model to win. It tells us which part of the comparison we have actually measured.

Now add the work around the model

Suppose Model A’s outputs take 20 hours of review across the trial, while Model B’s take ten. At an assumed loaded labour cost of £30 an hour, review adds £600 and £300 respectively.

Including model charges and review, A costs £604.80 for 800 accepted tasks, or £0.756 each. B costs £308.40 for 950, or about £0.325 each. The ranking reverses because review effort dominates this particular example.

These remain partial costs. They exclude integration, hosting, retrieval, support and the separate handling of tasks that never reached acceptance. If those rejected tasks must be completed manually, the business case needs that fallback cost as well. Dividing by accepted tasks already assigns the trial’s failed-attempt costs to its useful output; it does not account for future work that has not yet happened.

Keep price, quality and capacity visible

The FinOps Foundation distinguishes resource-efficiency measures from business unit measures. For this decision, token cost describes resource consumption; cost per accepted task connects that consumption to an outcome.

Use both. A rising token bill may reflect more successful work, longer inputs or unnecessary retries. Without the underlying measures, the same increase can be mistaken for either growth or waste.

Acceptance must also mean the same thing for both models. Faster review is not a saving if reviewers have quietly lowered the standard. The AI productivity business case needs the same treatment: count the complete workflow, not just the moment an answer appears.

Test the assumptions that can change the decision

Before committing, vary the inputs most likely to move: document length, output length, retry frequency, accepted-task rate and review time. Use a plausible busy period as well as an average one. If a small change in review effort reverses the winner, that assumption deserves more measurement than a minor difference in token price.

For zero accepted tasks, report cost per accepted task as unavailable, not zero. The spend still exists; the useful output does not.

A good model comparison ends with an explanation of what drives the result. That makes the decision revisable when prices, workloads or quality change, rather than tying it to a rate card that was never the whole cost.