The AI model metrics that actually matter
By Imran Sheikh · 2026-11-12 · Models and tools · 5 min read
Key points
- Public benchmarks shortlist models; your own eval picks one.
- Price per task beats price per token as a cost measure.
- Twenty real examples with clear pass rules is enough to start.
Model launches arrive with a wall of numbers. Scores, prices, context sizes, speed charts. Most of them are true, and most of them don’t answer the question a product team has, which is whether the model will do this job well enough, fast enough, at a price that works.
This post walks through the metrics we actually check, what each one tells you, and what it hides. Then it shows how to run a small eval of your own, which settles more decisions than any leaderboard.
The metrics at a glance
| Metric | What it tells you | What it hides |
|---|---|---|
| Benchmark index (e.g. Artificial Analysis) | Broad ability on hard, multi-step tasks | How the model does on your specific task |
| Human preference ranking (LMArena) | Which answers people prefer in blind votes | Accuracy on facts; votes can favour style |
| Coding benchmarks and WebDev votes | Ability to write and fix real code | Fit with your stack and conventions |
| Time to first token | The pause a user feels before anything appears | Total time for long answers |
| Output speed | How fast long answers stream | Thinking time before the answer |
| Price per 1M input and output tokens | The rate card | Thinking tokens, retries, tokenizer size |
| Context window | How much the model can read in one call | How well it uses the middle of a long input |
| Tool use and agent support | Whether it can call your functions and run multi-step work | How often it picks the wrong tool |
Quality benchmarks
Two kinds of quality signal are worth knowing. The first is a test-suite index. Artificial Analysis publishes an Intelligence Index, now at version 5.0.1, built from nine evaluations. It weights four areas equally at 25% each: knowledge work, coding, science and general knowledge. It is a good way to see roughly where a model sits. It also reports scores per effort setting. That alone is useful, because the same model can move ten points or more between medium and max effort.
The second is human preference. LMArena shows people two anonymous answers and asks which is better. Its text and WebDev leaderboards often disagree with each other, and that’s the point. As of November 11, 2026, a Gemini model leads the text board while Claude models hold the top two WebDev spots. If your product is mostly writing, look at text. If it’s mostly building interfaces, look at WebDev.
Use both to make a shortlist of two or three. Don’t use either to make the final call.
Latency
Latency is the metric users feel directly. Split it into two numbers. Time to first token is the wait before text appears. Output speed is how fast it streams after that. A chat reply can feel fine with a slower first token if it streams quickly. A one-word classification lives or dies on the first number.
Modern models think before they answer, and that thinking happens before the first visible token. Artificial Analysis’s figures show how much this matters. As of November 2026, GPT-6 Luna takes 1.74 seconds to its first answer token with reasoning off and 12.74 seconds at high effort. Same model, very different product. When you compare latency, compare at the effort setting you plan to ship.
Cost per million tokens
Every vendor publishes a price per million input tokens and per million output tokens. As of November 2026, Claude Opus 5.5 lists at $4 and $20. GPT-6.1 Sol and Claude Sonnet 5.5 both list at $2 and $10. Those rates are a starting point. Four things move the real number.
- Thinking tokens are billed as output, so higher effort costs more per answer.
- Tokenizers differ. Anthropic says its current tokenizer counts the same text as about 30% more tokens than the one before it. The same paragraph costs a different number of tokens on each vendor.
- Long prompts can cost more per token. OpenAI raises flagship prices above 272K input tokens. Claude Haiku 5.5 raises its rate above 100K.
- Caching and batch cut cost. Anthropic and OpenAI both offer 50% off batch work, and cached input is a fraction of the normal rate.
So the number that matters is cost per completed task. Run ten real tasks, add up the bill, divide by ten. Artificial Analysis reports a cost-per-task figure for its own suite, which is a helpful sanity check, but your tasks are the ones that count.
Context window and tool use
Context windows have mostly converged. As of November 2026, the current Claude, GPT-6 and Gemini 4 models all list about one million tokens. Anthropic estimates that 1M tokens holds roughly 555,000 words on its current tokenizer. That is enough to paste in a large codebase or a stack of research interviews. With every flagship near the same size, two other questions decide more. Does the model find the right detail in the middle of all that text? And do you want to pay to send it every time?
Tool use is the metric that matters most for agents. Can the model call your functions with the right arguments, read the result and decide what to do next? All three vendors support tool use on their current models. Coding benchmarks and agent tasks inside the Artificial Analysis index are the closest public proxy. The real test is your own tools, because a model can be great at code and still misuse a badly named function.
How to run your own small eval
Anthropic’s guide to building evals gives three principles we agree with. Make the test match your real task, edge cases included. Automate the grading where you can. And prefer more questions with simple automatic grading over a few hand-graded ones. Here is the small version we use.
- Write 20 to 50 real inputs from your product, including the awkward ones: empty fields, typos, other languages, angry users.
- For each, write what a pass looks like. Exact answers where possible, a short rule where not.
- Grade automatically: exact match for labels, a format check for JSON, and a second model with a yes or no question for tone or safety.
- Run two or three shortlisted models at the effort level you’d ship. Record pass rate, median latency and total cost.
- Read the failures yourself. The pattern in the misses tells you more than the score.
- Keep the set. Re-run it when a new model ships or when you change the prompt.
How we choose
Our order of questions is simple. Does it pass our eval? Is it fast enough at the effort level that passed? What does a completed task cost at next month’s prices? Benchmarks get a model onto the list. Our own twenty examples decide whether it stays. When two models tie, we pick the cheaper one and keep the other as a fallback.