The AI model metrics that actually matter

By Imran Sheikh · 2026-11-12 · Models and tools · 5 min read

Key points

  • Public benchmarks shortlist models; your own eval picks one.
  • Price per task beats price per token as a cost measure.
  • Twenty real examples with clear pass rules is enough to start.

Model launches arrive with a wall of numbers. Scores, prices, context sizes, speed charts. Most of them are true, and most of them don’t answer the question a product team has, which is whether the model will do this job well enough, fast enough, at a price that works.

This post walks through the metrics we actually check, what each one tells you, and what it hides. Then it shows how to run a small eval of your own, which settles more decisions than any leaderboard.

The metrics at a glance

What each common metric is good for, and where it stops being useful.
MetricWhat it tells youWhat it hides
Benchmark index (e.g. Artificial Analysis)Broad ability on hard, multi-step tasksHow the model does on your specific task
Human preference ranking (LMArena)Which answers people prefer in blind votesAccuracy on facts; votes can favour style
Coding benchmarks and WebDev votesAbility to write and fix real codeFit with your stack and conventions
Time to first tokenThe pause a user feels before anything appearsTotal time for long answers
Output speedHow fast long answers streamThinking time before the answer
Price per 1M input and output tokensThe rate cardThinking tokens, retries, tokenizer size
Context windowHow much the model can read in one callHow well it uses the middle of a long input
Tool use and agent supportWhether it can call your functions and run multi-step workHow often it picks the wrong tool

Quality benchmarks

Two kinds of quality signal are worth knowing. The first is a test-suite index. Artificial Analysis publishes an Intelligence Index, now at version 5.0.1, built from nine evaluations. It weights four areas equally at 25% each: knowledge work, coding, science and general knowledge. It is a good way to see roughly where a model sits. It also reports scores per effort setting. That alone is useful, because the same model can move ten points or more between medium and max effort.

The second is human preference. LMArena shows people two anonymous answers and asks which is better. Its text and WebDev leaderboards often disagree with each other, and that’s the point. As of November 11, 2026, a Gemini model leads the text board while Claude models hold the top two WebDev spots. If your product is mostly writing, look at text. If it’s mostly building interfaces, look at WebDev.

Use both to make a shortlist of two or three. Don’t use either to make the final call.

Latency

Latency is the metric users feel directly. Split it into two numbers. Time to first token is the wait before text appears. Output speed is how fast it streams after that. A chat reply can feel fine with a slower first token if it streams quickly. A one-word classification lives or dies on the first number.

Modern models think before they answer, and that thinking happens before the first visible token. Artificial Analysis’s figures show how much this matters. As of November 2026, GPT-6 Luna takes 1.74 seconds to its first answer token with reasoning off and 12.74 seconds at high effort. Same model, very different product. When you compare latency, compare at the effort setting you plan to ship.

Cost per million tokens

Every vendor publishes a price per million input tokens and per million output tokens. As of November 2026, Claude Opus 5.5 lists at $4 and $20. GPT-6.1 Sol and Claude Sonnet 5.5 both list at $2 and $10. Those rates are a starting point. Four things move the real number.

  • Thinking tokens are billed as output, so higher effort costs more per answer.
  • Tokenizers differ. Anthropic says its current tokenizer counts the same text as about 30% more tokens than the one before it. The same paragraph costs a different number of tokens on each vendor.
  • Long prompts can cost more per token. OpenAI raises flagship prices above 272K input tokens. Claude Haiku 5.5 raises its rate above 100K.
  • Caching and batch cut cost. Anthropic and OpenAI both offer 50% off batch work, and cached input is a fraction of the normal rate.

So the number that matters is cost per completed task. Run ten real tasks, add up the bill, divide by ten. Artificial Analysis reports a cost-per-task figure for its own suite, which is a helpful sanity check, but your tasks are the ones that count.

Context window and tool use

Context windows have mostly converged. As of November 2026, the current Claude, GPT-6 and Gemini 4 models all list about one million tokens. Anthropic estimates that 1M tokens holds roughly 555,000 words on its current tokenizer. That is enough to paste in a large codebase or a stack of research interviews. With every flagship near the same size, two other questions decide more. Does the model find the right detail in the middle of all that text? And do you want to pay to send it every time?

Tool use is the metric that matters most for agents. Can the model call your functions with the right arguments, read the result and decide what to do next? All three vendors support tool use on their current models. Coding benchmarks and agent tasks inside the Artificial Analysis index are the closest public proxy. The real test is your own tools, because a model can be great at code and still misuse a badly named function.

How to run your own small eval

Anthropic’s guide to building evals gives three principles we agree with. Make the test match your real task, edge cases included. Automate the grading where you can. And prefer more questions with simple automatic grading over a few hand-graded ones. Here is the small version we use.

  • Write 20 to 50 real inputs from your product, including the awkward ones: empty fields, typos, other languages, angry users.
  • For each, write what a pass looks like. Exact answers where possible, a short rule where not.
  • Grade automatically: exact match for labels, a format check for JSON, and a second model with a yes or no question for tone or safety.
  • Run two or three shortlisted models at the effort level you’d ship. Record pass rate, median latency and total cost.
  • Read the failures yourself. The pattern in the misses tells you more than the score.
  • Keep the set. Re-run it when a new model ships or when you change the prompt.

How we choose

Our order of questions is simple. Does it pass our eval? Is it fast enough at the effort level that passed? What does a completed task cost at next month’s prices? Benchmarks get a model onto the list. Our own twenty examples decide whether it stays. When two models tie, we pick the cheaper one and keep the other as a fallback.

Sources

  1. Artificial Analysis: Intelligence Index methodology
  2. Artificial Analysis: Model leaderboard
  3. LMArena: Text leaderboard
  4. LMArena: WebDev leaderboard
  5. Anthropic: Models overview
  6. Anthropic: Claude Haiku 5.5 overview
  7. Anthropic: Create strong empirical evaluations
  8. OpenAI: API pricing
  9. Google AI Studio: Gemini models

Journal · Book a call · Home