Picking a model when the budget is real and the data can't leave

The question benchmarks don't answer

Three new frontier models landed in about three weeks. Kimi K3 on July 16. GPT-5.6 Sol in preview on June 26. Claude Fable 5 already out and leading most boards. Each one shipped with a chart showing it winning.

None of those charts help me, because the question I have is not "which model is smartest." It's narrower and more annoying than that:

• The client has a fixed monthly budget, and it's usually smaller than you'd guess.
• Some clients' data cannot go to a US API. Not "shouldn't" — the owner will not agree to it, and that's the end of the conversation.
• The task is rarely "write a function." It's "read forty-page supplier PDFs and pull out the rates," or "sit inside a reconciliation flow and flag what doesn't add up."

Those three constraints decide it. The leaderboard is a filter I apply afterwards, if at all. Here's how I actually read the numbers.

Why most benchmark tables are worthless

Most scores are not comparable across labs.

When Anthropic publishes a SWE-bench number, they run it inside their own agent scaffold — their retry logic, their tool definitions, their prompts. OpenAI runs the same benchmark inside a different one. You aren't comparing two models. You're comparing two models plus two engineering teams' worth of scaffolding, and the scaffolding is worth a lot of points.

So when a vendor's chart shows their model ahead, some real part of that gap is just harness tuning. Fable 5's 80.3% on SWE-bench Pro is a vendor number in Anthropic's own scaffold, and independent aggregators dispute it. Its 95% on SWE-bench Verified is different — vals.ai confirmed that one independently, and I weight it accordingly.

The rule I use: if one organisation didn't measure all the models the same way, the comparison isn't one. That throws out most of what gets posted, and leaves Artificial Analysis, which runs everything itself under one setup.

What survives the filter

Same harness, all three models:

Read the first row and the last three together, because that's the actual trade.

Three points of index separate them. Sixty, fifty-nine, fifty-seven. On a normal task you will not feel that gap. You will absolutely feel the billing gap — Fable 5 runs about 3× K3's cost per completed task.

K3 took the frontend crown outright. Arena.ai's own numbers put it at 1,679 against Fable 5's 1,631 and Sol's 1,618, with a 76% pairwise win rate — humans blind-picking its output three times in four. It won six of seven frontend domains. First open-weight model to top that board.

GDPval flips the ranking back. That one scores realistic professional deliverables across 44 occupations, and Fable 5 leads with K3 last of the three. Longer horizon, more steps, different winner. (Neither tops it, incidentally — Opus 5 sits at 1,852.)

Two rows I deliberately left out. Terminal-Bench, because Sol's widely-quoted 91.9% is its high-effort Ultra mode and K3's 88.3% is base — base against base it's 88.8 to 88.3, a tie, and printing the other pair would be the exact dishonesty I'm complaining about. And SWE-bench, because I couldn't find same-harness numbers for all three, so a row there would have been decoration.

How the constraints actually resolve

When the data can't leave the building. This is the one that overrides everything. A business owner who won't send his supplier pricing to a US API is not making a technical objection you can argue him out of — for a lot of operations, the rate sheet is the competitive advantage. K3 is the only one of the three with open weights, which makes the entire rest of the table irrelevant. It's not the best model available. It's the best model that's allowed, and a model you can't use scores zero.

When the budget is the binding constraint. Also K3, for a different reason. Three points of index is not worth 3× the invoice on work that is mostly structured extraction and CRUD-shaped reasoning. Most of what I build is exactly that.

When it's long, multi-step agent work. Fable 5, and this is where paying 3× stops being silly. The failure mode that costs real money isn't a wrong function — it's step 14 quietly failing because step 3 was subtly wrong, and nobody noticing until the month-end numbers don't reconcile. GDPval and AA-Briefcase measure staying coherent over a long horizon, and Fable 5 leads both. On a reconciliation flow I want the model that doesn't drift.

When it lives in a terminal or a tool-calling loop. Sol. It tops the AA Coding Agent Index at 80 in the Codex harness, and it's half Fable 5's price.

When it's UI. K3, and it isn't close — see the frontend row.

The part no leaderboard shows

Simon Willison noted something about K3 worth more than any benchmark: it burned enough reasoning tokens to run 25 cents on a trivial prompt. Nothing in a leaderboard column tells you that. But on a system doing ten thousand small operations a day, per-call token appetite decides your bill far more than the sticker price per million does.

He makes the broader point too — that his informal test has "mostly severed" its connection to model quality, and doesn't touch agentic tool calling, which is the thing that matters most now.

That matches what I keep running into. Benchmarks narrow the field from ten models to two. They never pick the winner. The winner gets picked by running your own actual work through both — the real supplier PDF, the real reconciliation, the messy one with the handwritten note in the margin — and watching which one holds up.

Half the time the ranking inverts, because your workload doesn't look like anyone's benchmark. That's not a flaw in the benchmarks. It's just the difference between a leaderboard and a business.