A per-token price is not a price

I asked Gemini to name a colour. It said “Blue.”

The API reported the answer as one output token. The call was billed for a hundred and sixty nine.

Nothing errored. Nothing looked wrong. The response came back in the normal shape, with a number in the field every tutorial tells you to read, and that number was wrong by a factor of a hundred and sixty nine.

That is the interesting failure, and it is the reason I am writing this up.

What I was building

A small tool called Model Arena. One prompt, sent to Anthropic, OpenAI, Google and Groq at the same time, with the answers side by side and what each one cost.

The motivation was ordinary enough. Picking a model from a leaderboard tells you how it performs on somebody else’s task. I wanted to see how they did on mine, and I wanted the cost next to the answer rather than discovered a month later on an invoice.

I expected the interesting part to be answer quality. It was not. It was the money.

The same answer, at fifty-one times the price

All four were asked to explain an API rate limit in one sentence. All four produced about thirty words. I would struggle to rank the answers.

Look at the output column. Groq produced its thirty words in 50 tokens. Gemini used 452. The extra 400 are not answer. They are reasoning: thinking the model does before replying, billed as output, never shown to you.

This is not a big-model effect either. gpt-5-nano is the cheapest model OpenAI sells. Asked to name a colour, it spent 203 output tokens producing the word “Blue.”

Where it gets dangerous

Hidden reasoning being expensive is a known thing. You can price it, budget for it, decide it is worth it.

The problem is that the providers do not agree on where to report it, and one of them splits it into a field you have to know exists.

Google reports candidatesTokenCount, which is the visible answer, and thoughtsTokenCount, which is the reasoning. Their pricing page bills output “including thinking tokens”, so you owe for both. Read only the obvious field and you record one token for a call you were charged 169 for.

Groq has a different version of the same problem. It serves the OpenAI API on its own hardware, so an OpenAI client talks to it perfectly happily, but it reports token counts in its own x_groq envelope rather than the standard usage field. A client that only knows OpenAI will stream from Groq all day and never learn what anything cost. Not an error. Just a zero where a number should be.

OpenAI has a third variant. Token counts on a streamed response are opt-in. Without the right request parameter you get no usage data at all, and the call silently costs an unknown amount.

Three providers, three different ways to end up with a confident number that is not the number you will be billed.

What this means if you are measuring anything

Two things follow, and the second is worse than the first.

Token counts are not portable. If you compare two models on tokens burned, you are partly comparing how each provider chose to account for tokens. Within one model it is fine. Across providers it is only meaningful once you have verified you are reading the field they actually bill against.

Someone put this better than I did when I raised it publicly:

Token count is only a valid efficiency signal if you’ve verified you’re reading the same accounting field the provider actually bills against. Otherwise you’re measuring disclosure practices, not agent behavior.

— Manish Bhaktisagar

And the mapping is not stable over time. Reasoning tokens were once not a separate field anywhere. A provider adding a new token category does not break anything loudly. Your total just quietly gets smaller.

Which means the reconciliation itself needs a check. Otherwise you have moved the confidently wrong number one layer down rather than removed it.

Three rules I ended up building in

The code that came out of this is shaped by one idea repeated in three places: keep the measurement, derive the valuation, and never let a derived number harden into a fact.

An unknown price is never zero. If a model is not in the price table, the tool prints price unknown and the run total names what it excluded. A comparison table that quietly shows $0.00 for the one row nobody priced is worse than one showing nothing, because the reader cannot tell from the number that anything is missing.

A promotional price expires into unknown rather than staying wrong. One model in my table is on an introductory rate that ends on 31 August. That entry carries the date and stops answering afterwards. I deliberately did not encode the successor price. A stale price looks exactly like a current one at the moment you read the total, which is precisely too late.

The database stores tokens and has no cost column. Tokens are what the provider reported and they never change. A price is a number on a page somebody else edits. Store the money and last month’s history quietly becomes a set of figures that were true once, with nothing marking which. Store the tokens and any run can be repriced at today’s rates, which is what the tool does every time you load a saved run.

There is a test asserting the absence of that cost column. A comment gets deleted by someone being helpful. A failing test does not.

What I have not done

None of these figures has been reconciled against an actual invoice.

They are arithmetic over a price table I typed by hand, read off each provider’s own pricing page, with the date I checked recorded next to them. That is why the function is called estimate and not cost.

Until you diff your own counter against a real bill you have a plausible number rather than a verified one. And plausible is exactly the failure this whole piece is about.

The general version

For years the invoice did safety work that nobody designed. Automation that looped or retried or quietly went wrong cost money roughly in proportion to how wrong it went, and a lot of runaway processes got caught because somebody looked at a bill, not because an alert fired.

Inference is now cheap enough that the bill has stopped being loud. Something can misbehave for a week without ever becoming expensive enough for anyone to notice.

So the cost signal has to be built rather than inherited. And a cost signal that reads the wrong field is not a smaller version of having one. It is worse than nothing, because it produces a number that looks like measurement.

The bill is not the place to find out.

Methodology

Everything above came off real calls made on 3 August 2026 from a UK connection, using Model Arena. You can reproduce any of it with your own keys.

• One sample per model per prompt. Not averaged. Reasoning token counts vary noticeably run to run, so treat individual figures as illustrative and the pattern as the finding.
• max_tokens 1024, reasoning effort left at each provider’s default. Nothing was tuned to make any model look better or worse.
• Timings include network latency from a domestic connection, so they are not a benchmark of the providers’ own speed.
• Prices are list prices read off each provider’s own pricing page on the day, with no volume discounts, no batch rates and no caching.
• Nothing has been reconciled against an invoice. See above.

Prices used

Raw runs

“In one sentence, what is an API rate limit?” — the table at the top of this piece.

Four for four in 7.6 seconds. Sequentially it would have been 16.3. Total $0.005156.

“In one sentence, what is a Server-Sent Event?” — a second prompt, same pattern.

“Name one colour.” — where the discrepancy at the top came from. Gemini’s raw usageMetadata, verbatim:

{
  "promptTokenCount": 7,
  "candidatesTokenCount": 1,
  "thoughtsTokenCount": 168,
  "totalTokenCount": 176,
  "serviceTier": "standard"
}

The visible answer was the word “Blue.” On the same prompt, gpt-5-nano returned 203 output tokens and llama-3.3-70b-versatile returned 3.

The code

If you want to check the reasoning rather than the numbers, these are the parts that matter:

• providers/gemini.py — where thoughtsTokenCount is added to candidatesTokenCount, and why.
• providers/groq.py — the whole x_groq dialect, in about thirty lines.
• pricing.py — the price table, the expiry rule, and the refusal to return zero.
• storage.py — the schema with no cost column.
• tests/test_parsing.py — the test pinned to the 7 / 1 / 168 figures above.
• FINDINGS.md — the full technical write-up, including every wire-format difference between the four providers.

The tool is MIT licensed and runs on your own keys. If you reproduce this and get different numbers, I would genuinely like to know.