Don’t Trust the Benchmarks. Do Your Own Research.

A few days ago I gave a talk called "What does AI really cost?" a follow-up to the cost post in this series. I expected to spend the Q&A talking about token prices, subscriptions and context management. Instead, the slide that got the most questions was the one about benchmarks – or rather, about why I stopped trusting them and built my own. So this post is that slide, expanded: what the public leaderboards currently say, why I think they tell at most half the truth, and what happened when I let eight models implement the same 20 real feature tickets of an Angular component library and had four humans judge the results blind.

Quick recap for everyone joining late: this series started with which LLM is best for Angular development, continued with which app or harness to wrap around it, and then moved on to the harness setup and the skills that turn a capable model into a careful teammate. The summer update was the last time I ranked models – mostly on public benchmarks plus my own gut feeling. This post replaces the gut feeling with data. The cost side of the talk gets its own follow-up post soon, so I'll keep the money talk short here.

TL;DR: Two Leagues, No Middle Field

If you only have thirty seconds: on the public benchmarks, the frontier models from Anthropic, OpenAI, Google and Meta sit within a few points of each other. On my own benchmark – eight models, the same 20 real Angular tickets, four blind human reviews – they don't. Three models landed between 7.5 and 8.25 out of 10, the other five between 3.5 and 4.75, and nothing in between. All three top spots went to Anthropic: Opus 5.5 for quality, Sonnet 5.5 for value (94 % of the Opus score for a fifth of the price), Fable 5.1 right behind (why third? – one of the reviewers flagged too much refactoring instead of focusing on the ticket with a minimal changeset). The GPT models, Gemini, Grok and Haiku all ended up a league lower – for real feature work I would not use them for agentic coding. And if you pay API prices, the harness matters too: the same Sonnet 5.5 cost about half as much per ticket in OpenCode (and 60 % in Pi) as in Claude Code, and a bit cheaper than in GitHub Copilot.

As always in this series: this is one benchmark on one Angular component lib codebase, judged by four humans. It is not deterministic, it is not universal, and it might be outdated in a month. But it is real work, reviewed by the people who have to live with the result – and that, I'll argue, is worth more than any leaderboard you can easily find out there.

What the Public Benchmarks Say

Let me start where everyone starts: the leaderboards. My favorite source is still Artificial Analysis, for the reasons I gave in the first post of this series – independent, transparent about their methodology, and the only ones who consistently report cost and time next to the score. The three charts below are snapshots from October 25, 2026, the first two filtered down to the 14 models I care about, the third the release view with all effort variants of 11 of them; click any of them for the live version.

Intelligence Index

Artificial Analysis Intelligence Index v4.3.2 – snapshot from October 25, 2026, 14 of 708 models. The index combines ten evaluations, from agentic tasks (Terminal-Bench, GDPval) over coding (SciCode) to long-context and scientific reasoning.

The top of the chart is crowded: Opus 5.5 at 58, Sonnet 5.5 and Muse Wonder Full at 56, Kimi K3.1 Pro at 55, Fable 5.1, GPT-6 Astra and Gemini 4 Argon at 53, Muse Wonder at 52. Eight models within six points. If you only looked at this chart, you'd conclude that the frontier is a tie and you should simply pick the cheapest one.

Cost per Task

Weighted average cost in USD per Intelligence Index task, segmented by token type – snapshot from October 25, 2026. Lower is better.

This chart I like, because it measures the right unit: not dollars per million tokens, but dollars per solved task. A model that charges little per token but burns ten times the reasoning tokens is not cheap. And here the spread is enormous: GPT-6 Luna solves an index task for 7 cents, Fable 5.1 for $7.63 – a factor of 100. The Anthropic models are the three most expensive bars on the right. Sonnet 5.5 costs $5.46 per task, Opus 5.5 $5.98 – almost the same, because at "max with fallback" Sonnet thinks a lot. Max should never be used with Anthropic models for real work – in my experience it is way too much overthinking, and frankly bullshit. Keep that in mind when we get to my own numbers, where Sonnet ran at high effort and turned out dramatically cheaper.

Intelligence vs. Cost per Task

Intelligence Index vs. cost per task by model release, all reasoning and effort variants of each release connected – snapshot from October 25, 2026. The dotted line is the Pareto front, the green area the "most attractive quadrant".

Put the two together and you get the chart I keep coming back to. The Pareto front runs from GPT-6 Luna in the bottom left through MiMo and GPT-6.1 Sol to Kimi K3.1 Pro and Opus 5.5 in the top right. GPT-6.1 Sol sits inside the attractive quadrant: an index of 52 for well under a dollar per task. GPT-6 Astra scores the same 53 as Fable 5.1 for less than half the cost. And Anthropic's top variants all cost more than $5 per task.

So the public benchmarks, read honestly, say: Anthropic has no lead worth paying for. OpenAI, Meta and Kimi are right there at a fraction of the price. That was also the tone of my summer update – and it is completely wrong. At least for our kind of work and for our team. One caveat up front: four of the models that make the frontier look like a tie – Muse Wonder Full, Kimi K3.1 Pro, GPT-6 Astra and Gemini 4 Argon – were not part of my bench, so what follows is about the eight models I did run.

Why I Don't Trust Them (Anymore)

Before I show you my own results, let me explain why I stopped treating leaderboards as a basis for model decisions. Not because the people running them are sloppy – Artificial Analysis in particular does careful work – but because of what a benchmark is.

A benchmark is a fixed set of tasks with a fixed way to score them. That has three consequences:

1. Contamination. The tasks are public, so sooner or later they end up in the training data. The most famous example is SWE-bench Verified, for years the coding benchmark. In February 2026, OpenAI stopped reporting it: their own models reproduced original patches from nothing but the task ID, and a manual audit found that most remaining "failures" were flawed tests, not model limits. The research community has by now written a whole taxonomy of contamination – and of the mitigations that each new generation of models defeats.
2. Goodhart's law. "When a measure becomes a target, it ceases to be a good measure." Every lab has a team whose job it is to make the next model look good on the handful of benchmarks that get quoted in launch posts. The Leaderboard Illusion, a NeurIPS 2025 paper, documented how this works on Chatbot Arena: private testing of dozens of variants, selective disclosure of the best one, and preferential data access for a few providers. The paper is about one arena, but the incentive applies to every public benchmark – and the more popular the benchmark, the stronger the pull.
3. Wrong task, wrong judge. Even an uncontaminated, ungamed benchmark measures what is easy to measure: did the test pass, did the answer match. My daily reality is different. A feature can pass every test and still be unmergeable; it can implement the ticket text and miss its intent; it can work and leave behind code nobody wants to own. No public benchmark runs on an enterprise Angular codebase with a strict style guide, a design system and an opinionated review culture – and none of them has a human reviewer at the end.

That third point is the one I can do something about. If the benchmark doesn't measure my work, I have to measure my work.

My Own Benchmark: Eight Models, 20 Tickets, Four Human Juries

So in early October 2026 I built one. The subject is an enterprise Angular component library I developed: a publishable library with secondary entry points, a demo app that serves as its living documentation, a strict style guide, an AGENTS.md with the repo rules, and e2e tests with Cypress. In short, exactly the kind of codebase this series has been about.

The Setup

• Eight models, each in one uninterrupted run: Opus 5.5, Sonnet 5.5, Fable 5.1 and Haiku 5.5 from Anthropic, GPT 6.1 Sol and GPT 6.0 Luna from OpenAI, Gemini 3.8 from Google, and Grok 4.7 from xAI. Each at the reasoning effort I'd normally use for it – high, the small models Luna and Haiku even on xhigh – with one deliberate exception: Fable at medium, to see whether the most intelligent model could get away with less thinking.
• The same 20 tickets, real feature tickets from the backlog, frozen in one file with a binding "Decisions" block per ticket that settled the open questions upfront. Tickets 1 to 16 extend existing components (select, chip area, navigation tree, tabs, checkbox, temporal editors, splitter, …), 17 to 20 create new ones (status badge, AG Grid theme, progress indicator, breadcrumb).
• The same starting commit, the same prompt, the same repo rules. The prompt asked for one commit per ticket, a self-review in a fresh context folded in by amend, and the full gate – lint, library build, app build, the touched e2e specs – green before every commit. No questions allowed: where the ticket was silent, the model had to decide and record the assumption.
• Each model in its own harness, the one its subscription ties it to: the Claude models in Claude Code, the GPT models in Codex CLI, Gemini in Antigravity, Grok in Cursor. That is the setup you get on a subscription anyway, so model and harness were benchmarked as the pair you'd actually use.
• What was recorded: the API cost per run (all eight were paid through my private subscriptions, but priced at list prices for comparison – all prices in this post are USD at API list prices, as the charts show them) and the active wall-clock time. Both stayed sealed until the scores were in.

The Four Juries

Public benchmarks measure what is easy to measure, so the judges here are the people who live with the result. Four reviewers, one question each, every one of them blind: the eight runs were deployed as numbered copies of the demo app and pushed as lettered branches, and none of the three colleagues knew which number or letter was which model. I did, which is why my own review needed a trick – see below.

• Ticket intent – the colleague who wrote the tickets checked on five of them whether each implementation does what he meant, not what the text said.
• Functional testing – a second colleague tested 19 of the 20 tickets on the deployed demos, with mouse and keyboard, hunting for bugs and regressions.
• Merge readiness – a third colleague, a senior developer, reviewed four sampled commits of each branch like a merge request: would this go into develop as it is?
• Code quality – that was me: all 20 tickets of every branch, read against the style guide and the repo rules. Is this code I'd be glad to own a year from now? Since I had started the runs and knew which model was behind each letter, I tricked myself into not knowing: I reviewed numbered copies of the branches and looked up the mapping only after my scores were in.

Every reviewer scored every run on the same scale from 0 to 10. The combined score is the plain average of the four medians – no weighting, no cleverness. Between them, the four of us spent well over 20 hours on the reviews. Remember that number; it comes back later.

The Ranking Table

The eight runs ranked by the average of the four reviews, with the four individual scores, the API cost per ticket and in total, and the active time of each run. * Gemini 3.8 stopped once because its context was full and had to be restarted; it then stopped three more times to ask a question. ** Implemented only 19 of the 20 tickets (there was an instruction to skip the ticket if certain red flags like failing e2e tests appear).

The first thing that jumps out is the split I already spoiled in the TL;DR: two leagues, no middle field. Opus 5.5 (8.25), Sonnet 5.5 (7.75) and Fable 5.1 (7.5) form the top league; Gemini 3.8 (4.75), Grok 4.7 and GPT 6.1 Sol (4.25), GPT 6.0 Luna (3.75) and Haiku 5.5 (3.5) the lower one. Nothing lands between 4.75 and 7.5. On this codebase, a model either delivers work the reviewers can build on, or it does not.

The second thing: the top three are all Claude models, and the three top places are close. That was the biggest surprise for me – not that Opus won, but that Sonnet and Fable ended up within three quarters of a point of it. Fable was even marked down a little by the merge-readiness reviewer for its tendency to refactor and improve things outside the ticket's scope: helpful on a solo project, extra review work in a team. And remember that Fable ran at medium effort, the other two at high – it almost got away with it.

The third thing: the two GPT models fell short, and so did everyone else. GPT 6.1 Sol and GPT 6.0 Luna both skipped one of the 20 tickets. Gemini 3.8 had to be restarted and stopped three more times to ask questions although the prompt forbade it. Grok 4.7 was the slowest run at 9 hours 10 minutes and the second most expensive, and still only shared fifth place. Time did not buy quality, and neither did price – Grok costs more per ticket than Opus.

And the fourth: Haiku 5.5 came last. The vendor alone is not the whole story. Anthropic's smallest model produced the smallest diffs, skipped features silently and broke an existing behavior. A cheap run that has to be redone is not cheap – more on that in the next post.

The Score Profiles

One chart per run: the four reviewers' scores on four axes – ticket intent, functional testing, merge readiness and code quality.

The radar charts explain why the leagues exist. The three top runs are big, balanced diamonds: no axis below 6, most at 7 or above. Opus scores 9 on both merge readiness and code quality. Sonnet got a perfect 10 for merge readiness – the reviewer's clear favorite, "very well rounded and reliable, no nonsense, many tests, sensible refactoring, initiative of its own." Fable leads on ticket intent with an 8: it understood what the ticket author meant better than anyone else, and then did a bit more than he asked for.

The lower league is where the profiles fall apart, each in its own way. Haiku 5.5 got a 6 for code quality, because it wrote the smallest diffs and the single best refactor of the whole bench (one table replacing 200 lines of switch statements), but a 1 for functional testing and merge readiness, because it quietly left features out. GPT 6.0 Luna got a 7 for ticket intent – it understood the tickets fine – with 2 and 3 on the code axes: "few and simple code changes, but the features often do not work." And then there is the one profile that deserves its own section.

The Gemini Case: Passes the Tests, Fails the Review

Look at Gemini 3.8. Functional testing: 9 – the best score of all eight runs, better than Opus. The tester's verdict: "provided the best solutions on average; only on rare occasions another implementation beat this one." If my benchmark had stopped there – if it had been a public benchmark, which scores exactly this: does it work, does the test pass – Gemini 3.8 would have won. At $1.31 per ticket, cheaper than Sonnet and a fraction of Opus and Fable. The perfect Pareto winner.

Now look at the other three axes. Ticket intent: a 6 overall, but on one of the five sampled tickets the ticket author gave it a 0 – no documentation, no showcase, nothing he could even check – which is not what he asked for. Merge readiness: 3 – "incomplete", one of the four reviewed commits a functional failure. And code quality: 1, the worst of all eight runs. My own notes from the blind review: the code looks disciplined at first glance, but it stacks defensive guards, dead ?? [] fallbacks and the same expression pasted eight times, and it introduced two regressions in existing components. On top of that it was the only run that needed babysitting: it stopped once because its context was full and had to be restarted, and then stopped three more times to ask questions, although the prompt forbade it.

So the same run is the best functional implementer and the worst engineer of the bench. The features work, and nobody on the team would merge them. Every one of these commits would come back from review – rewritten by a human, or redone by a better model – and that is where the cheap price per ticket evaporates. Functional testing 9 is what a benchmark measures. Code quality 1 is what you pay for afterwards.

This is the pattern a single-number benchmark can't show you, and it is why I wrote this post. A model that is optimized to pass tests will pass tests. Whether it writes code you want to own only a human review on your own codebase can tell – and in this case the human review turned the apparent winner into what is potentially the most expensive model of the bench – once you add the human hours to rewrite its code.

Score vs. Costs

Every run by its API cost per ticket (logarithmic) and its average score, with the Pareto front and the attractive quadrant – cheap and good. * restarted, ** 19 of 20 tickets.

Now price and quality in one picture, in the same style as the Artificial Analysis chart above – and this is where my numbers and theirs diverge. The Pareto front has a steep step in the middle: from GPT 6.1 Sol at $0.71 and 4.25 points through Gemini 3.8 at $1.31 and 4.75 straight up to Sonnet 5.5 at $1.46 and 7.75 points. For 15 cents more per ticket you move from the lower league to the upper one. From there the line flattens: Opus 5.5 buys another half point for $7.20 per ticket, and Fable 5.1, at $11.18 per ticket, sits below the line – more expensive and slightly worse than Opus in this run, at medium effort.

Three practical takeaways from this chart, in the order I'd make the decision:

• Opus 5.5 is currently the best choice for implementing features. The highest score, and both cheaper and faster than Fable: $7.20 against $11.18 per ticket, 3 h 34 min against 5 h 10 min for all 20 tickets.
• Sonnet 5.5 is the best value by far. 94 % of the Opus score for a fifth of its cost. It is the only run in the attractive quadrant, and the one I'd recommend to every team that pays API prices.
• Fable 5.1 stays my favorite for everything that isn't implementation. Planning, design, reviews, the mutual understanding between model and human – it is still the most intelligent of the three in my eyes. When it implements, tell it explicitly to stay within the ticket, and consider high effort.

And where does GPT 6.1 Sol still fit? At $0.71 per ticket and 2 h 53 min it is far cheaper and faster than the top three. I'd use it for very small, well-defined tasks, for prototypes and experiments, and as the second opinion from another provider in a review – three Claude models share the same blind spots. For anything that ends up in develop, I'd pay the 75 cents.

Even the most expensive run is cheap in absolute terms, by the way. All eight runs together, 158 tickets, would cost about $613 at API list prices. Without AI I'd estimate each of these tickets at two to four hours of implementation work – at a conservative €50 per hour that is €100 to €200 per ticket, roughly 15 to 25 times the cost of the Opus run. The token bill is not the expensive part of agentic engineering. But that is the next post.

The Verdict: Use the Anthropic Models

On the public benchmarks these three are tied with half a dozen others; on human review they are a league ahead. That difference is exactly the difference between models that are good and models that look good on benchmarks – so for agentic coding in the frontend, use the Anthropic models.

That brings me to the rough ranking I showed in the talk – my personal order of providers for agentic coding, as of Q4 2026:

1. Anthropic – Opus 5.5, Sonnet 5.5, Fable 5.1. The top league, see above.
2. OpenAI – GPT-6 Astra and GPT-6.1 Sol. I left Astra out of the bench on purpose: from daily use I don't rate it as a coding model, so I spent the review hours on Sol and Luna instead – but for reviews and research it's my go-to. Sol is my pick for small, cheap tasks and second opinions.
3. Meta – Muse Wonder Full is the one model I'd pay a separate subscription for next. Everything below I wouldn't touch for agentic coding at all – yes, they might be useful, but I don't want to use them 😏
4. Google, xAI or the Chinese open-weight models – Gemini 3.8 and Grok 4.7 both landed in the lower league. DeepSeek 4.1, GLM 5.3, Kimi 3.1 and MiMo 2.6 are consistently three to six months behind the frontier, and that gap has been remarkably stable – no catching up in sight. If they catch up, we re-evaluate everything.
5. Local models – on the hardware most of us have, usually at least one more step down. They make sense for exactly one reason: when your data must not leave the company. Then they are the only option, and the data privacy post applies.

Two models on that list weren't in our human bench at all: Muse Wonder Full and Gemini 4 Argon were both released after the runs. On the public charts they sit right at the top – but this bench only judges the eight models I ran. It took four people more than 20 hours to judge them, which is exactly why I cannot repeat it whenever a new model drops.

My plan is to rerun it roughly every quarter, with the new releases in the lineup. In between, the Artificial Analysis charts above are the best update on capability and cost you'll get – just read them as what they are. The bench can't tell you anything about Gemini 4 Argon; my own first days with it can: for agentic coding the way I do it, Google has not caught up.

Hot take: if somebody tells you that Gemini 4 Argon has caught up with the frontier, it's worth asking what they measured. On benchmarks and first impressions it has; in my first days of real agentic coding with it, not yet. The same goes for claims about local models, by the way. But now let's look at the harnesses for a bit.

The Harness Matters (If You Pay API Prices)

One more result from the same tickets, because it surprised me almost as much as the model ranking. On a subscription, the harness is a given: Claude Code for the Claude models, Codex for the GPT models, and so on – that is how the eight model runs above were set up. But as soon as you pay API prices, you can run any model in any harness, and then the choice is yours. So after the eight model runs, I ran a harness follow-up: the same 20 tickets, the same prompt, one fixed model – Sonnet 5.5 at medium effort – in four different coding harnesses, in two rounds. In the first round one agent worked through all 20 tickets itself; in the second, the agent handed every ticket to a fresh subagent and only orchestrated. No human review this time, only cost and time. Within a round, nothing changed between the runs but the wrapper: the system prompt, the tool definitions, the skills and rules it loads, and how it runs its subagents.

Harness	Cost per ticket	Total cost	Time
OpenCode	$0.84	$16.71	2 h 23 min
Pi	$0.98	$19.65	3 h 19 min
GitHub Copilot	$1.02	$20.31	2 h 35 min
Claude Code	$1.63	$32.52	3 h 40 min

Sonnet 5.5 (medium), 20 tickets, every figure the average of the two rounds per harness. Claude Code went directly to the Anthropic API, the other three through the GitHub Copilot API; all tokens priced at Anthropic list prices.

Same model, same task, twice the bill. OpenCode and Pi cost roughly half of Claude Code per ticket, with GitHub Copilot in between. The reason is context: every turn of an agentic session sends the whole context again, and Claude Code's context is simply bigger – a longer system prompt, more tools, subagents. The two rounds show where the gap comes from: with one continuous agent, Claude Code ($22) and GitHub Copilot ($21) were level, OpenCode ($15) and Pi ($16) a quarter cheaper. With one subagent per ticket, Claude Code's bill nearly doubled to $43 for the 20 tickets, Pi rose to $23, OpenCode to $18, and Copilot even fell to $19.

Two caveats before you switch. First, this follow-up measured cost and time, not quality – Sonnet at high effort scored 7.75 in the model bench, and medium effort was not reviewed. Second, if you work on a Claude or OpenAI subscription, none of this matters much: you pay a flat price, and Claude Code's subagents, hooks and /loop are worth the context they cost. It matters when you pay API prices – which, in my experience, is the normal case in the enterprise. Then my low-budget setup is Sonnet 5.5 (medium) in OpenCode or Pi, at a cost per ticket close to GPT 6.1 Sol and with a model whose high-effort sibling sits in the upper league – verify the quality of medium on your own tickets first. How the four token types, caching and context management drive that bill is the next post.

Run Your Own Bench

You don't need eight models and 20 tickets. What you need is your own codebase, your own reviewers and a fair setup:

1. Pick three to five real tickets from your backlog – not demos, not greenfield. Freeze the text and settle the open questions upfront, so every model works from the same requirement.
2. Give two or three candidate models the same prompt, the same starting commit and the same rules. Let each one run through all tickets on its own, with the full gate green before every commit.
3. Let the people who normally review the work score the results – blind. Numbered deployments, lettered branches, one question per reviewer. The ticket author checks intent, a tester checks behavior, a reviewer checks mergeability, someone reads the code.
4. Count the hours around each run, not just the tokens: restarts, questions, skipped tickets, out-of-scope changes, review rounds. That is where the cost is.

A few days of this tell you more about your codebase than any leaderboard – and the setup is reusable the next time a vendor claims the top spot.

Agentic Engineering Workshop

In our Agentic Engineering Workshop we build your setup hands-on: your style guide, your AGENTS.md, and a skills catalogue tuned to your team's rules instead of mine – and then put it to work on spec-first workflows, code review, testing, and brownfield refactoring in real Angular codebases. Which model and which harness to use is one of the first decisions we make there, and after this post you know how I'd make it. English or German, as always.

• 🤖 Agentic Engineering Workshop – 2 days (or 3 half-days online), remote or in-house, English or German, for advanced Angular developers – next public date (in English): November 30 to December 2, 2026

Conclusion

Don't trust the benchmarks. Not because they lie, but because they measure the wrong task with the wrong judge, and because every lab is optimizing for them. The public charts show the frontier as a tie; four humans reviewing real Angular work on my codebase show two leagues with nothing in between.

So, for Q4 2026, my recommendation in one line each:

• Models: the Anthropic models. Opus 5.5 when quality matters, Sonnet 5.5 when cost matters, Fable 5.1 for planning, design and reviews. GPT-6.1 Sol for small, well-defined tasks and as the cheap second opinion from another provider. Astra for architecture and planning reviews and computer use tasks.
• Harness, on a subscription: whatever you're productive in – for me that's Claude Code for Fable and Opus, and Codex for Astra and Sol.
• Harness, on API prices: OpenCode or Pi, with Sonnet 5.5 at medium effort. Same model, half the bill – medium was not reviewed for quality, so check it on your own tickets.
• Orchestrator app, the GUI on top of those coding agents: T3 Code, still and by far. Nothing I know of competes with it in functionality and developer experience, and the team behind it really knows what they are doing. Why exactly it is the best option deserves its own post – that one is planned for right after the cost follow-up.

And do your own research: your codebase, your tickets, your reviewers. It costs a few days and a few hundred euros in tokens, and it is the only benchmark that measures what you actually pay for.

Next up: what agentic coding really costs – cost per merged pull request, why subscriptions are fair and the API isn't, the four token types and why caching is the reason any of this is affordable, and the context management that cuts the bill. Coming soon.

Thank you for reading 🙏 this blog post was written by Alexander Thalhammer. For feedback, remarks or questions, please reach out to me ❤️