Model Arena

Picking a model from a leaderboard tells you how it performs on somebody else's task. Model Arena runs your prompt against Anthropic, OpenAI, Google and Groq at the same time, and puts the four answers next to each other with tokens, timing and real money.

I built it because I wanted the cost next to the answer rather than discovered a month later on an invoice. I expected the interesting part to be answer quality. It was the money.

What it does

model-arena "why is the sky blue?"        # every provider at once
model-arena -m openai:gpt-5 "why is..."   # pin one
model-arena --history                     # past runs
model-arena --show 12                     # reprint one, repriced at today's rates
model-arena --serve                       # browse them in a browser

Four providers behind one interface, asked concurrently, with normalised token counts and honest cost. Every run is saved so two models can be diffed on the same prompt later.

What building it taught

Most provider diversity is one wire format with dialects. All four open a streaming POST and read Server-Sent Events. Only three things vary: where to send it, what the body looks like, and how to read a chunk. So those three are the entire interface, and the transport is written once. Adding a fifth provider means a request builder and a parser, and touching no networking code.

Groq is the clearest case. It serves the OpenAI API on its own hardware, so the whole provider is four constants and one override.

The token accounting is where the bodies are buried. Anthropic reports input and output at opposite ends of the stream. OpenAI sends both once, but only if you ask. Gemini repeats a running total on every chunk, and splits reasoning into a separate field. Summing usage as it arrives is correct for two of them and would multiply Gemini's number by the chunk count.

That produced the finding worth writing up on its own: a call that reported one output token had actually been billed for a hundred and sixty nine. The rest is in A per-token price is not a price.

Only the transport is async. Building a request and parsing a chunk stayed ordinary functions, because neither waits for anything. It is tempting when learning asyncio to make everything async def, but then await stops carrying information. Kept narrow, every await in the codebase marks something that genuinely blocks, and there is exactly one.

Four providers asked together answer in the time of the slowest, not the sum: 5.1 seconds against 14.4. The claim is measured rather than asserted, with a fake network built from httpx.MockTransport so the test would fail if the calls stopped overlapping.

One idea, repeated in three places. Keep the measurement, derive the valuation, and never let a derived number harden into a fact.

• An unknown price reports as unknown, never as zero, and the run total names what it excluded.
• A promotional price carries its expiry and lapses into unknown rather than staying wrong.
• The database stores tokens and deliberately has no cost column, so any run can be repriced. There is a test asserting the absence of that column, because a comment gets deleted by someone being helpful and a failing test does not.

Honest limitations

Nothing has been reconciled against an actual invoice. The prices are a table I typed by hand off each provider's own pricing page, which is why the function is called estimate rather than cost.

It also has no model catalogue, on purpose. Model ids date within weeks and the point is to try things newer than the tool, so a model is just a string handed to the provider.

Where to look

103 tests, none of which touch the network. Ruff and mypy strict clean. FINDINGS.md is the full technical write-up, including every wire-format difference between the four providers.

MIT licensed, runs on your own keys.