{"id":35151,"date":"2026-10-29T15:00:26","date_gmt":"2026-10-29T14:00:26","guid":{"rendered":"https:\/\/www.angulararchitects.io\/?p=35151"},"modified":"2026-10-29T15:19:51","modified_gmt":"2026-10-29T14:19:51","slug":"ae-human-bench","status":"publish","type":"post","link":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/","title":{"rendered":"Don&#8217;t Trust the Benchmarks. Do Your Own Research."},"content":{"rendered":"<div class=\"wp-post-series-box series-agentic-engineering wp-post-series-box--expandable\">\n\t\t\t<input id=\"collapsible-series-agentic-engineering6ae372673f635\" class=\"wp-post-series-box__toggle_checkbox\" type=\"checkbox\">\n\t\n\t<label\n\t\tclass=\"wp-post-series-box__label\"\n\t\t\t\t\tfor=\"collapsible-series-agentic-engineering6ae372673f635\"\n\t\t\ttabindex=\"0\"\n\t\t\t\t>\n\t\t<p class=\"wp-post-series-box__name wp-post-series-name\">\n\t\t\tThis is post 10 of 10 in the series <em>&ldquo;Agentic Engineering&rdquo;<\/em>\t\t<\/p>\n\t\t\t<\/label>\n\n\t\t\t<div class=\"wp-post-series-box__posts\">\n\t\t\t<ol>\n\t\t\t\t\t\t\t\t\t<li><a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/best-llms-for-angular\/\">Agentic Engineering: Which LLM Is Best for Angular Development?<\/a><\/li>\n\t\t\t\t\t\t\t\t\t<li><a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ai-apps-harnesses-for-angular\/\">Agentic Engineering: Which App\/Harness Is Best for Angular Development?<\/a><\/li>\n\t\t\t\t\t\t\t\t\t<li><a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ai-costs-for-angular\/\">Agentic Engineering: What Does AI Coding Really Cost?<\/a><\/li>\n\t\t\t\t\t\t\t\t\t<li><a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ai-data-privacy-for-angular\/\">Agentic Engineering: What Do AI Coding Tools Do With Your Code?<\/a><\/li>\n\t\t\t\t\t\t\t\t\t<li><a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ai-personal-verdict\/\">Agentic Verdict: What&#8217;s the Best Solution for You?<\/a><\/li>\n\t\t\t\t\t\t\t\t\t<li><a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ai-next-gen-model\/\">Anthropic Just Released the First Next-Gen Model to the Public<\/a><\/li>\n\t\t\t\t\t\t\t\t\t<li><a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-harness-setup-for-angular\/\">Agentic Engineering: How to Set Up Your Harness for Angular Development<\/a><\/li>\n\t\t\t\t\t\t\t\t\t<li><a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-summer-2026-update-for-angular\/\">Agentic Engineering: Summer 2026 Update<\/a><\/li>\n\t\t\t\t\t\t\t\t\t<li><a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-skills-for-angular\/\">Agentic Engineering: Skills for Angular Development<\/a><\/li>\n\t\t\t\t\t\t\t\t\t<li><span class=\"wp-post-series-box__current\">Don&#8217;t Trust the Benchmarks. Do Your Own Research.<\/span><\/li>\n\t\t\t\t\t\t\t<\/ol>\n\t\t<\/div>\n\t<\/div>\n<p>A few days ago I gave a talk called <em>&quot;What does AI really cost?&quot;<\/em> a follow-up to <a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ai-costs-for-angular\/\">the cost post<\/a> in this series. I expected to spend the Q&amp;A talking about token prices, subscriptions and context management. Instead, the slide that got the most questions was the one about <strong>benchmarks<\/strong> \u2013 or rather, about why I stopped trusting them and built my own. So this post is that slide, expanded: what the public leaderboards currently say, why I think they tell at most half the truth, and what happened when I let eight models implement the same 20 real feature tickets of an <em>Angular<\/em> component library and had four humans judge the results <strong><em>blind<\/em><\/strong>.<\/p>\n<p>Quick recap for everyone joining late: this series started with <a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/best-llms-for-angular\/\">which LLM is best for <em>Angular<\/em> development<\/a>, continued with <a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ai-apps-harnesses-for-angular\/\">which app or harness to wrap around it<\/a>, and then moved on to the <a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-harness-setup-for-angular\/\">harness setup<\/a> and the <a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-skills-for-angular\/\">skills<\/a> that turn a capable model into a careful teammate. The <a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-summer-2026-update-for-angular\/\">summer update<\/a> was the last time I ranked models \u2013 mostly on public benchmarks plus my own gut feeling. This post replaces the gut feeling with data. The cost side of the talk gets its own follow-up post soon, so I'll keep the money talk short here.<\/p>\n<style>\n.single-post-content {\n  font-size: 0.8em;\n  line-height: 1.6;\n}\n.single-post-content a {\n  color: #4183c4;\n  text-decoration: underline;\n}\n.single-post-content a:hover {\n  color: #2a6cb0;\n}\n.single-post-content :not(pre) > code {\n  padding: 0.15em 0.4em;\n  border-radius: 3px;\n  background: rgba(128, 128, 128, 0.15);\n  font-size: 0.9em;\n}\n<\/style>\n<p><!-- Only needed for posts with tables --><\/p>\n<style>\n.single-post-content table {\n  width: 100%;\n  margin: 1.5em 0;\n  border-collapse: collapse;\n  font-size: 0.9em;\n  line-height: 1.4;\n}\n.single-post-content th,\n.single-post-content td {\n  padding: 0.5em 0.75em;\n  border-bottom: 1px solid rgba(128, 128, 128, 0.4);\n  text-align: left;\n  vertical-align: top;\n}\n.single-post-content th {\n  border-bottom-width: 2px;\n}\n.single-post-content td:first-child {\n  white-space: nowrap;\n}\n@media (max-width: 767px) {\n  .single-post-content table {\n    font-size: 1em;\n    line-height: 1.5;\n  }\n  .single-post-content table,\n  .single-post-content tbody,\n  .single-post-content tr,\n  .single-post-content td {\n    display: block;\n    width: 100%;\n  }\n  .single-post-content thead {\n    display: none;\n  }\n  .single-post-content tr {\n    padding: 0.75em 0;\n    border-bottom: 1px solid rgba(128, 128, 128, 0.4);\n  }\n  .single-post-content td {\n    padding: 0.15em 0;\n    border-bottom: 0;\n    overflow-wrap: anywhere;\n  }\n  .single-post-content td:first-child {\n    white-space: normal;\n    font-size: 1.1em;\n  }\n}\n<\/style>\n<h2>TL;DR: Two Leagues, No Middle Field<\/h2>\n<p>If you only have thirty seconds: on the public benchmarks, the frontier models from Anthropic, OpenAI, Google and Meta sit within a few points of each other. On <strong>my own benchmark<\/strong> \u2013 eight models, the same 20 real <em>Angular<\/em> tickets, four blind human reviews \u2013 they don't. Three models landed between 7.5 and 8.25 out of 10, the other five between 3.5 and 4.75, and nothing in between. All three top spots went to Anthropic: <strong>Opus 5.5<\/strong> for quality, <strong>Sonnet 5.5<\/strong> for value (94 % of the Opus score for a fifth of the price), <strong>Fable 5.1<\/strong> right behind (why third? \u2013 one of the reviewers flagged too much refactoring instead of focusing on the ticket with a minimal changeset). The GPT models, Gemini, Grok and Haiku all ended up a league lower \u2013 for real feature work I would <strong>not use them for agentic coding<\/strong>. And if you pay API prices, the harness matters too: the same Sonnet 5.5 cost about half as much per ticket in <strong>OpenCode<\/strong> (and 60 % in <strong>Pi<\/strong>) as in <strong>Claude Code<\/strong>, and a bit cheaper than in GitHub Copilot.<\/p>\n<p>As always in this series: this is one benchmark on one <em>Angular<\/em> component lib codebase, judged by <strong>four humans<\/strong>. It is not deterministic, it is not universal, and it might be outdated in a month. But it is real work, reviewed by the people who have to live with the result \u2013 and that, I'll argue, is worth <strong>more than any leaderboard<\/strong> you can easily find out there.<\/p>\n<h2>What the Public Benchmarks Say<\/h2>\n<p>Let me start where everyone starts: the leaderboards. My favorite source is still <a href=\"https:\/\/artificialanalysis.ai\/\">Artificial Analysis<\/a>, for the reasons I gave in the <a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/best-llms-for-angular\/\">first post of this series<\/a> \u2013 independent, transparent about their <a href=\"https:\/\/artificialanalysis.ai\/methodology\">methodology<\/a>, and the only ones who consistently report cost and time next to the score. The three charts below are snapshots from October 25, 2026, the first two filtered down to the 14 models I care about, the third the release view with all effort variants of 11 of them; click any of them for the <a href=\"https:\/\/artificialanalysis.ai\/?models=claude-opus-5-5%2Cclaude-sonnet-5-5%2Cmuse-wonder-full%2Cclaude-fable-5-1%2Cgpt-6-astra%2Cgemini-4-argon%2Cgemini-3-8-flash%2Cgrok-4-7%2Cclaude-haiku-5-5%2Cgpt-6-luna%2Ckimi-k3-1-pro%2Cmuse-wonder%2Cdeepseek-v4-1-flash-1017%2Cmimo-v2-6-pro\">live version<\/a>.<\/p>\n<h3>Intelligence Index<\/h3>\n<p><a href=\"https:\/\/artificialanalysis.ai\/?models=claude-opus-5-5%2Cclaude-sonnet-5-5%2Cmuse-wonder-full%2Cclaude-fable-5-1%2Cgpt-6-astra%2Cgemini-4-argon%2Cgemini-3-8-flash%2Cgrok-4-7%2Cclaude-haiku-5-5%2Cgpt-6-luna%2Ckimi-k3-1-pro%2Cmuse-wonder%2Cdeepseek-v4-1-flash-1017%2Cmimo-v2-6-pro\"><img decoding=\"async\" src=\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-aa-intelligence-index.png\" alt=\"Artificial Analysis Intelligence Index, October 2026\" \/><\/a><\/p>\n<p>Artificial Analysis Intelligence Index v4.3.2 \u2013 snapshot from October 25, 2026, 14 of 708 models. The index combines ten evaluations, from agentic tasks (Terminal-Bench, GDPval) over coding (SciCode) to long-context and scientific reasoning.<\/p>\n<p>The top of the chart is crowded: Opus 5.5 at 58, Sonnet 5.5 and Muse Wonder Full at 56, Kimi K3.1 Pro at 55, Fable 5.1, GPT-6 Astra and Gemini 4 Argon at 53, Muse Wonder at 52. Eight models within six points. If you only looked at this chart, you'd conclude that the frontier is a tie and you should simply pick the cheapest one.<\/p>\n<h3>Cost per Task<\/h3>\n<p><a href=\"https:\/\/artificialanalysis.ai\/?models=claude-opus-5-5%2Cclaude-sonnet-5-5%2Cmuse-wonder-full%2Cclaude-fable-5-1%2Cgpt-6-astra%2Cgemini-4-argon%2Cgemini-3-8-flash%2Cgrok-4-7%2Cclaude-haiku-5-5%2Cgpt-6-luna%2Ckimi-k3-1-pro%2Cmuse-wonder%2Cdeepseek-v4-1-flash-1017%2Cmimo-v2-6-pro\"><img decoding=\"async\" src=\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-aa-cost-per-task.png\" alt=\"Artificial Analysis Cost per Intelligence Index Task, October 2026\" \/><\/a><\/p>\n<p>Weighted average cost in USD per Intelligence Index task, segmented by token type \u2013 snapshot from October 25, 2026. Lower is better.<\/p>\n<p>This chart I like, because it measures the right unit: not dollars per million tokens, but <strong>dollars per solved task<\/strong>. A model that charges little per token but burns ten times the reasoning tokens is not cheap. And here the spread is enormous: GPT-6 Luna solves an index task for 7 cents, Fable 5.1 for $7.63 \u2013 a factor of 100. The Anthropic models are the three most expensive bars on the right. Sonnet 5.5 costs $5.46 per task, Opus 5.5 $5.98 \u2013 almost the same, because at &quot;max with fallback&quot; Sonnet thinks a lot. Max should never be used with Anthropic models for real work \u2013 in my experience it is way too much overthinking, and frankly bullshit. Keep that in mind when we get to my own numbers, where Sonnet ran at <em>high<\/em> effort and turned out dramatically cheaper.<\/p>\n<h3>Intelligence vs. Cost per Task<\/h3>\n<p><a href=\"https:\/\/artificialanalysis.ai\/?models=claude-opus-5-5%2Cclaude-sonnet-5-5%2Cmuse-wonder-full%2Cclaude-fable-5-1%2Cgpt-6-astra%2Cgemini-4-argon%2Cgemini-3-8-flash%2Cgrok-4-7%2Cclaude-haiku-5-5%2Cgpt-6-luna%2Ckimi-k3-1-pro%2Cmuse-wonder%2Cdeepseek-v4-1-flash-1017%2Cmimo-v2-6-pro\"><img decoding=\"async\" src=\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-aa-index-vs-cost.png\" alt=\"Artificial Analysis Intelligence Index vs. Cost per Task, October 2026\" \/><\/a><\/p>\n<p>Intelligence Index vs. cost per task by model release, all reasoning and effort variants of each release connected \u2013 snapshot from October 25, 2026. The dotted line is the Pareto front, the green area the &quot;most attractive quadrant&quot;.<\/p>\n<p>Put the two together and you get the chart I keep coming back to. The Pareto front runs from GPT-6 Luna in the bottom left through MiMo and GPT-6.1 Sol to Kimi K3.1 Pro and Opus 5.5 in the top right. GPT-6.1 Sol sits inside the attractive quadrant: an index of 52 for well under a dollar per task. GPT-6 Astra scores the same 53 as Fable 5.1 for less than half the cost. And Anthropic's top variants all cost more than $5 per task.<\/p>\n<p>So the public benchmarks, read honestly, say: <strong>Anthropic has no lead worth paying for.<\/strong> OpenAI, Meta and Kimi are right there at a fraction of the price. That was also the tone of my summer update \u2013 and it is <strong>completely wrong<\/strong>. At least for our kind of work and for our team. One caveat up front: four of the models that make the frontier look like a tie \u2013 Muse Wonder Full, Kimi K3.1 Pro, GPT-6 Astra and Gemini 4 Argon \u2013 were not part of my bench, so what follows is about the eight models I did run.<\/p>\n<h2>Why I Don't Trust Them (Anymore)<\/h2>\n<p>Before I show you my own results, let me explain why I stopped treating leaderboards as a basis for model decisions. Not because the people running them are sloppy \u2013 Artificial Analysis in particular does careful work \u2013 but because of what a benchmark <em>is<\/em>.<\/p>\n<p>A benchmark is a fixed set of tasks with a fixed way to score them. That has three consequences:<\/p>\n<ol>\n<li><strong>Contamination.<\/strong> The tasks are public, so sooner or later they end up in the training data. The most famous example is <a href=\"https:\/\/www.swebench.com\/\">SWE-bench Verified<\/a>, for years <em>the<\/em> coding benchmark. In February 2026, OpenAI <a href=\"https:\/\/openai.com\/index\/why-we-no-longer-evaluate-swe-bench-verified\/\">stopped reporting it<\/a>: their own models reproduced original patches from nothing but the task ID, and a manual audit found that most remaining &quot;failures&quot; were flawed tests, not model limits. The research community has by now written a whole <a href=\"https:\/\/arxiv.org\/abs\/2608.29463\">taxonomy of contamination<\/a> \u2013 and of the mitigations that each new generation of models defeats.<\/li>\n<li><strong>Goodhart's law.<\/strong> &quot;When a measure becomes a target, it ceases to be a good measure.&quot; Every lab has a team whose job it is to make the next model look good on the handful of benchmarks that get quoted in launch posts. <a href=\"https:\/\/arxiv.org\/abs\/2504.20879\">The Leaderboard Illusion<\/a>, a NeurIPS 2025 paper, documented how this works on Chatbot Arena: private testing of dozens of variants, selective disclosure of the best one, and preferential data access for a few providers. The paper is about one arena, but the incentive applies to every public benchmark \u2013 and the more popular the benchmark, the stronger the pull.<\/li>\n<li><strong>Wrong task, wrong judge.<\/strong> Even an uncontaminated, ungamed benchmark measures what is easy to measure: did the test pass, did the answer match. My daily reality is different. A feature can pass every test and still be unmergeable; it can implement the ticket text and miss its intent; it can work and leave behind code nobody wants to own. No public benchmark runs on an enterprise <em>Angular<\/em> codebase with a strict style guide, a design system and an opinionated review culture \u2013 and none of them has a human reviewer at the end.<\/li>\n<\/ol>\n<p>That third point is the one I can do something about. If the benchmark doesn't measure my work, I have to measure my work.<\/p>\n<h2>My Own Benchmark: Eight Models, 20 Tickets, Four Human Juries<\/h2>\n<p>So in early October 2026 I built one. The subject is an enterprise <em>Angular<\/em> component library I developed: a publishable library with secondary entry points, a demo app that serves as its living documentation, a strict style guide, an <code>AGENTS.md<\/code> with the repo rules, and e2e tests with Cypress. In short, exactly the kind of codebase this series has been about.<\/p>\n<h3>The Setup<\/h3>\n<ul>\n<li><strong>Eight models<\/strong>, each in one uninterrupted run: Opus 5.5, Sonnet 5.5, Fable 5.1 and Haiku 5.5 from Anthropic, GPT 6.1 Sol and GPT 6.0 Luna from OpenAI, Gemini 3.8 from Google, and Grok 4.7 from xAI. Each at the reasoning effort I'd normally use for it \u2013 high, the small models Luna and Haiku even on xhigh \u2013 with one deliberate exception: Fable at medium, to see whether the most intelligent model could get away with less thinking.<\/li>\n<li><strong>The same 20 tickets<\/strong>, real feature tickets from the backlog, frozen in one file with a binding &quot;Decisions&quot; block per ticket that settled the open questions upfront. Tickets 1 to 16 extend existing components (select, chip area, navigation tree, tabs, checkbox, temporal editors, splitter, \u2026), 17 to 20 create new ones (status badge, AG Grid theme, progress indicator, breadcrumb).<\/li>\n<li><strong>The same starting commit<\/strong>, the same prompt, the same repo rules. The prompt asked for one commit per ticket, a self-review in a fresh context folded in by amend, and the full gate \u2013 lint, library build, app build, the touched e2e specs \u2013 green before every commit. No questions allowed: where the ticket was silent, the model had to decide and record the assumption.<\/li>\n<li><strong>Each model in its own harness<\/strong>, the one its subscription ties it to: the Claude models in Claude Code, the GPT models in Codex CLI, Gemini in Antigravity, Grok in Cursor. That is the setup you get on a subscription anyway, so model and harness were benchmarked as the pair you'd actually use.<\/li>\n<li><strong>What was recorded<\/strong>: the API cost per run (all eight were paid through my private subscriptions, but priced at list prices for comparison \u2013 all prices in this post are USD at API list prices, as the charts show them) and the active wall-clock time. Both stayed sealed until the scores were in.<\/li>\n<\/ul>\n<h3>The Four Juries<\/h3>\n<p>Public benchmarks measure what is easy to measure, so the judges here are the people who live with the result. Four reviewers, one question each, every one of them blind: the eight runs were deployed as numbered copies of the demo app and pushed as lettered branches, and none of the three colleagues knew which number or letter was which model. I did, which is why my own review needed a trick \u2013 see below.<\/p>\n<ul>\n<li><strong>Ticket intent<\/strong> \u2013 the colleague who wrote the tickets checked on five of them whether each implementation does what he <em>meant<\/em>, not what the text said.<\/li>\n<li><strong>Functional testing<\/strong> \u2013 a second colleague tested 19 of the 20 tickets on the deployed demos, with mouse and keyboard, hunting for bugs and regressions.<\/li>\n<li><strong>Merge readiness<\/strong> \u2013 a third colleague, a senior developer, reviewed four sampled commits of each branch like a merge request: would this go into <code>develop<\/code> as it is?<\/li>\n<li><strong>Code quality<\/strong> \u2013 that was me: all 20 tickets of every branch, read against the style guide and the repo rules. Is this code I'd be glad to own a year from now? Since I had started the runs and knew which model was behind each letter, I tricked myself into not knowing: I reviewed numbered copies of the branches and looked up the mapping only after my scores were in.<\/li>\n<\/ul>\n<p>Every reviewer scored every run on the same scale from 0 to 10. The combined score is the plain average of the four medians \u2013 no weighting, no cleverness. Between them, the four of us spent <strong>well over 20 hours<\/strong> on the reviews. Remember that number; it comes back later.<\/p>\n<h3>The Ranking Table<\/h3>\n<p><img decoding=\"async\" src=\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-ranking-table.png\" alt=\"Ranking table of the eight model runs\" \/><\/p>\n<p>The eight runs ranked by the average of the four reviews, with the four individual scores, the API cost per ticket and in total, and the active time of each run. * Gemini 3.8 stopped once because its context was full and had to be restarted; it then stopped three more times to ask a question. ** Implemented only 19 of the 20 tickets (there was an instruction to skip the ticket if certain red flags like failing e2e tests appear).<\/p>\n<p>The first thing that jumps out is the split I already spoiled in the TL;DR: <strong>two leagues, no middle field.<\/strong> Opus 5.5 (8.25), Sonnet 5.5 (7.75) and Fable 5.1 (7.5) form the top league; Gemini 3.8 (4.75), Grok 4.7 and GPT 6.1 Sol (4.25), GPT 6.0 Luna (3.75) and Haiku 5.5 (3.5) the lower one. Nothing lands between 4.75 and 7.5. On this codebase, a model either delivers work the reviewers can build on, or it does not.<\/p>\n<p>The second thing: <strong>the top three are all Claude models, and the three top places are close.<\/strong> That was the biggest surprise for me \u2013 not that Opus won, but that Sonnet and Fable ended up within three quarters of a point of it. Fable was even marked down a little by the merge-readiness reviewer for its tendency to refactor and improve things outside the ticket's scope: helpful on a solo project, extra review work in a team. And remember that Fable ran at <em>medium<\/em> effort, the other two at <em>high<\/em> \u2013 it almost got away with it.<\/p>\n<p>The third thing: <strong>the two GPT models fell short, and so did everyone else.<\/strong> GPT 6.1 Sol and GPT 6.0 Luna both skipped one of the 20 tickets. Gemini 3.8 had to be restarted and stopped three more times to ask questions although the prompt forbade it. Grok 4.7 was the slowest run at 9 hours 10 minutes and the second most expensive, and still only shared fifth place. Time did not buy quality, and neither did price \u2013 Grok costs more per ticket than Opus.<\/p>\n<p>And the fourth: <strong>Haiku 5.5 came last.<\/strong> The vendor alone is not the whole story. Anthropic's smallest model produced the smallest diffs, skipped features silently and broke an existing behavior. A cheap run that has to be redone is not cheap \u2013 more on that in the next post.<\/p>\n<h3>The Score Profiles<\/h3>\n<p><img decoding=\"async\" src=\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-score-profiles.png\" alt=\"Score profiles of the eight model runs\" \/><\/p>\n<p>One chart per run: the four reviewers' scores on four axes \u2013 ticket intent, functional testing, merge readiness and code quality.<\/p>\n<p>The radar charts explain <em>why<\/em> the leagues exist. The three top runs are big, balanced diamonds: no axis below 6, most at 7 or above. Opus scores 9 on both merge readiness and code quality. Sonnet got a perfect 10 for merge readiness \u2013 the reviewer's clear favorite, &quot;very well rounded and reliable, no nonsense, many tests, sensible refactoring, initiative of its own.&quot; Fable leads on ticket intent with an 8: it understood what the ticket author meant better than anyone else, and then did a bit more than he asked for.<\/p>\n<p>The lower league is where the profiles fall apart, each in its own way. <strong>Haiku 5.5<\/strong> got a 6 for code quality, because it wrote the smallest diffs and the single best refactor of the whole bench (one table replacing 200 lines of switch statements), but a 1 for functional testing and merge readiness, because it quietly left features out. <strong>GPT 6.0 Luna<\/strong> got a 7 for ticket intent \u2013 it understood the tickets fine \u2013 with 2 and 3 on the code axes: &quot;few and simple code changes, but the features often do not work.&quot; And then there is the one profile that deserves its own section.<\/p>\n<h3>The Gemini Case: Passes the Tests, Fails the Review<\/h3>\n<p>Look at <strong>Gemini 3.8<\/strong>. Functional testing: <strong>9<\/strong> \u2013 the best score of all eight runs, better than Opus. The tester's verdict: &quot;provided the best solutions on average; only on rare occasions another implementation beat this one.&quot; If my benchmark had stopped there \u2013 if it had been a public benchmark, which scores exactly this: does it work, does the test pass \u2013 Gemini 3.8 would have <strong>won<\/strong>. At $1.31 per ticket, cheaper than Sonnet and a fraction of Opus and Fable. The perfect Pareto winner.<\/p>\n<p>Now look at the other three axes. Ticket intent: a 6 overall, but on one of the five sampled tickets the ticket author gave it a <strong>0<\/strong> \u2013 no documentation, no showcase, nothing he could even check \u2013 which is not what he asked for. Merge readiness: <strong>3<\/strong> \u2013 &quot;incomplete&quot;, one of the four reviewed commits a functional failure. And code quality: <strong>1<\/strong>, the worst of all eight runs. My own notes from the blind review: the code looks disciplined at first glance, but it stacks defensive guards, dead <code>?? []<\/code> fallbacks and the same expression pasted eight times, and it introduced two regressions in existing components. On top of that it was the only run that needed babysitting: it stopped once because its context was full and had to be restarted, and then stopped three more times to ask questions, although the prompt forbade it.<\/p>\n<p>So the same run is the <strong>best functional implementer and the worst engineer<\/strong> of the bench. The features work, and nobody on the team would merge them. Every one of these commits would come back from review \u2013 rewritten by a human, or redone by a better model \u2013 and that is where the cheap price per ticket evaporates. Functional testing 9 is what a benchmark measures. Code quality 1 is what you pay for afterwards.<\/p>\n<p>This is the pattern a single-number benchmark can't show you, and it is why I wrote this post. A model that is optimized to pass tests will pass tests. Whether it writes code you want to own only a human review on your own codebase can tell \u2013 and in this case the human review turned the apparent winner into what is potentially the most expensive model of the bench \u2013 once you add the human hours to rewrite its code.<\/p>\n<h3>Score vs. Costs<\/h3>\n<p><img decoding=\"async\" src=\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-score-vs-cost.png\" alt=\"Score vs. cost per ticket of the eight model runs\" \/><\/p>\n<p>Every run by its API cost per ticket (logarithmic) and its average score, with the Pareto front and the attractive quadrant \u2013 cheap and good. * restarted, ** 19 of 20 tickets.<\/p>\n<p>Now price and quality in one picture, in the same style as the Artificial Analysis chart above \u2013 and this is where my numbers and theirs diverge. The Pareto front has a steep step in the middle: from GPT 6.1 Sol at $0.71 and 4.25 points through Gemini 3.8 at $1.31 and 4.75 straight up to Sonnet 5.5 at $1.46 and 7.75 points. For 15 cents more per ticket you move from the lower league to the upper one. From there the line flattens: Opus 5.5 buys another half point for $7.20 per ticket, and Fable 5.1, at $11.18 per ticket, sits below the line \u2013 more expensive and slightly worse than Opus in this run, at medium effort.<\/p>\n<p>Three practical takeaways from this chart, in the order I'd make the decision:<\/p>\n<ul>\n<li><strong>Opus 5.5 is currently the best choice for implementing features.<\/strong> The highest score, and both cheaper and faster than Fable: $7.20 against $11.18 per ticket, 3 h 34 min against 5 h 10 min for all 20 tickets.<\/li>\n<li><strong>Sonnet 5.5 is the best value by far.<\/strong> 94 % of the Opus score for a fifth of its cost. It is the only run in the attractive quadrant, and the one I'd recommend to every team that pays API prices.<\/li>\n<li><strong>Fable 5.1 stays my favorite for everything that isn't implementation.<\/strong> Planning, design, reviews, the mutual understanding between model and human \u2013 it is still the most intelligent of the three in my eyes. When it implements, tell it explicitly to stay within the ticket, and consider high effort.<\/li>\n<\/ul>\n<p>And where does GPT 6.1 Sol still fit? At $0.71 per ticket and 2 h 53 min it is far cheaper and faster than the top three. I'd use it for very small, well-defined tasks, for prototypes and experiments, and as the second opinion from another provider in a review \u2013 three Claude models share the same blind spots. For anything that ends up in <code>develop<\/code>, I'd pay the 75 cents.<\/p>\n<p>Even the most expensive run is cheap in absolute terms, by the way. All eight runs together, 158 tickets, would cost about <strong>$613<\/strong> at API list prices. Without AI I'd estimate each of these tickets at two to four hours of implementation work \u2013 at a conservative \u20ac50 per hour that is \u20ac100 to \u20ac200 per ticket, roughly 15 to 25 times the cost of the Opus run. The token bill is not the expensive part of agentic engineering. But that is the next post.<\/p>\n<h3>The Verdict: Use the Anthropic Models<\/h3>\n<p>On the public benchmarks these three are tied with half a dozen others; on human review they are a league ahead. That difference is exactly the difference between models that are good and models that look good on benchmarks \u2013 so for agentic coding in the frontend, <strong>use the Anthropic models.<\/strong><\/p>\n<p>That brings me to the rough ranking I showed in the talk \u2013 my personal order of providers for agentic coding, as of Q4 2026:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-provider-ranking.png\" alt=\"My rough ranking of providers for agentic coding, Q4 2026\" \/><\/p>\n<ol>\n<li><strong>Anthropic<\/strong> \u2013 Opus 5.5, Sonnet 5.5, Fable 5.1. The top league, see above.<\/li>\n<li><strong>OpenAI<\/strong> \u2013 GPT-6 Astra and GPT-6.1 Sol. I left Astra out of the bench on purpose: from daily use I don't rate it as a coding model, so I spent the review hours on Sol and Luna instead \u2013 but for reviews and research it's my go-to. Sol is my pick for small, cheap tasks and second opinions.<\/li>\n<li><strong>Meta<\/strong> \u2013 Muse Wonder Full is the one model I'd pay a separate subscription for next. Everything below I wouldn't touch for agentic coding at all \u2013 yes, they might be useful, but I don't want to use them \ud83d\ude0f<\/li>\n<li><strong>Google, xAI or the Chinese open-weight models<\/strong> \u2013 Gemini 3.8 and Grok 4.7 both landed in the lower league. DeepSeek 4.1, GLM 5.3, Kimi 3.1 and MiMo 2.6 are consistently three to six months behind the frontier, and that gap has been remarkably stable \u2013 no catching up in sight. If they catch up, we re-evaluate everything.<\/li>\n<li><strong>Local models<\/strong> \u2013 on the hardware most of us have, usually at least one more step down. They make sense for exactly one reason: when your data must not leave the company. Then they are the only option, and the <a href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ai-data-privacy-for-angular\/\">data privacy post<\/a> applies.<\/li>\n<\/ol>\n<p>Two models on that list weren't in our human bench at all: <strong>Muse Wonder Full<\/strong> and <strong>Gemini 4 Argon<\/strong> were both released after the runs. On the public charts they sit right at the top \u2013 but this bench only judges the eight models I ran. It took four people more than 20 hours to judge them, which is exactly why I cannot repeat it whenever a new model drops.<\/p>\n<p>My plan is to rerun it roughly every quarter, with the new releases in the lineup. In between, the Artificial Analysis charts above are the best update on capability and cost you'll get \u2013 just read them as what they are. The bench can't tell you anything about Gemini 4 Argon; my own first days with it can: for agentic coding the way I do it, Google has <strong>not caught up<\/strong>.<\/p>\n<p><strong>Hot take:<\/strong> if somebody tells you that Gemini 4 Argon has caught up with the frontier, it's worth asking what they measured. On benchmarks and first impressions it has; in my first days of real agentic coding with it, not yet. The same goes for claims about local models, by the way. But now let's look at the harnesses for a bit.<\/p>\n<h2>The Harness Matters (If You Pay API Prices)<\/h2>\n<p>One more result from the same tickets, because it surprised me almost as much as the model ranking. On a subscription, the harness is a given: Claude Code for the Claude models, Codex for the GPT models, and so on \u2013 that is how the eight model runs above were set up. But as soon as you pay <strong>API prices<\/strong>, you can run any model in any harness, and then the choice is yours. So after the eight model runs, I ran a <strong>harness follow-up<\/strong>: the same 20 tickets, the same prompt, one fixed model \u2013 Sonnet 5.5 at medium effort \u2013 in four different coding harnesses, in two rounds. In the first round one agent worked through all 20 tickets itself; in the second, the agent handed every ticket to a fresh subagent and only orchestrated. No human review this time, only cost and time. Within a round, nothing changed between the runs but the wrapper: the system prompt, the tool definitions, the skills and rules it loads, and how it runs its subagents.<\/p>\n<table>\n<thead>\n<tr>\n<th>Harness<\/th>\n<th style=\"text-align: right;\">Cost per ticket<\/th>\n<th style=\"text-align: right;\">Total cost<\/th>\n<th>Time<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><strong>OpenCode<\/strong><\/td>\n<td style=\"text-align: right;\">$0.84<\/td>\n<td style=\"text-align: right;\">$16.71<\/td>\n<td>2 h 23 min<\/td>\n<\/tr>\n<tr>\n<td><strong>Pi<\/strong><\/td>\n<td style=\"text-align: right;\">$0.98<\/td>\n<td style=\"text-align: right;\">$19.65<\/td>\n<td>3 h 19 min<\/td>\n<\/tr>\n<tr>\n<td><strong>GitHub Copilot<\/strong><\/td>\n<td style=\"text-align: right;\">$1.02<\/td>\n<td style=\"text-align: right;\">$20.31<\/td>\n<td>2 h 35 min<\/td>\n<\/tr>\n<tr>\n<td><strong>Claude Code<\/strong><\/td>\n<td style=\"text-align: right;\">$1.63<\/td>\n<td style=\"text-align: right;\">$32.52<\/td>\n<td>3 h 40 min<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Sonnet 5.5 (medium), 20 tickets, every figure the average of the two rounds per harness. Claude Code went directly to the Anthropic API, the other three through the GitHub Copilot API; all tokens priced at Anthropic list prices.<\/p>\n<p><strong>Same model, same task, twice the bill.<\/strong> <a href=\"https:\/\/opencode.ai\/\">OpenCode<\/a> and <a href=\"https:\/\/github.com\/badlogic\/pi-mono\">Pi<\/a> cost roughly half of <a href=\"https:\/\/www.anthropic.com\/product\/claude-code\">Claude Code<\/a> per ticket, with GitHub Copilot in between. The reason is context: every turn of an agentic session sends the whole context again, and Claude Code's context is simply bigger \u2013 a longer system prompt, more tools, subagents. The two rounds show where the gap comes from: with one continuous agent, Claude Code ($22) and GitHub Copilot ($21) were level, OpenCode ($15) and Pi ($16) a quarter cheaper. With one subagent per ticket, Claude Code's bill nearly doubled to $43 for the 20 tickets, Pi rose to $23, OpenCode to $18, and Copilot even fell to $19.<\/p>\n<p>Two caveats before you switch. First, this follow-up measured cost and time, not quality \u2013 Sonnet at high effort scored 7.75 in the model bench, and medium effort was not reviewed. Second, if you work on a Claude or OpenAI <strong>subscription<\/strong>, none of this matters much: you pay a flat price, and Claude Code's subagents, hooks and <code>\/loop<\/code> are worth the context they cost. It matters when you pay <strong>API prices<\/strong> \u2013 which, in my experience, is the normal case in the enterprise. Then my low-budget setup is Sonnet 5.5 (medium) in OpenCode or Pi, at a cost per ticket close to GPT 6.1 Sol and with a model whose high-effort sibling sits in the upper league \u2013 verify the quality of medium on your own tickets first. How the four token types, caching and context management drive that bill is the next post.<\/p>\n<h2>Run Your Own Bench<\/h2>\n<p>You don't need eight models and 20 tickets. What you need is your own codebase, your own reviewers and a fair setup:<\/p>\n<ol>\n<li>Pick <strong>three to five real tickets<\/strong> from your backlog \u2013 not demos, not greenfield. Freeze the text and settle the open questions upfront, so every model works from the same requirement.<\/li>\n<li>Give <strong>two or three candidate models<\/strong> the same prompt, the same starting commit and the same rules. Let each one run through all tickets on its own, with the full gate green before every commit.<\/li>\n<li>Let <strong>the people who normally review the work<\/strong> score the results \u2013 blind. Numbered deployments, lettered branches, one question per reviewer. The ticket author checks intent, a tester checks behavior, a reviewer checks mergeability, someone reads the code.<\/li>\n<li><strong>Count the hours around each run<\/strong>, not just the tokens: restarts, questions, skipped tickets, out-of-scope changes, review rounds. That is where the cost is.<\/li>\n<\/ol>\n<p>A few days of this tell you more about your codebase than any leaderboard \u2013 and the setup is reusable the next time a vendor claims the top spot.<\/p>\n<h2>Agentic Engineering Workshop<\/h2>\n<p>In our <strong>Agentic Engineering Workshop<\/strong> we build your setup hands-on: your style guide, your <code>AGENTS.md<\/code>, and a skills catalogue tuned to your team's rules instead of mine \u2013 and then put it to work on spec-first workflows, code review, testing, and brownfield refactoring in real <em>Angular<\/em> codebases. Which model and which harness to use is one of the first decisions we make there, and after this post you know how I'd make it. English or German, as always.<\/p>\n<ul>\n<li>\ud83e\udd16 <a href=\"https:\/\/www.angulararchitects.io\/en\/training\/agentic-engineering-von-vibe-coding-zu-professionellen-ki-gestuetzten-workflows\/\"><strong>Agentic Engineering Workshop<\/strong><\/a> \u2013 2 days (or 3 half-days online), remote or in-house, English or German, for advanced <em>Angular<\/em> developers \u2013 next public date (in English): <strong>November 30 to December 2, 2026<\/strong><\/li>\n<\/ul>\n<h2>Conclusion<\/h2>\n<p>Don't trust the benchmarks. Not because they lie, but because they measure the wrong task with the wrong judge, and because every lab is optimizing for them. The public charts show the frontier as a tie; four humans reviewing real <em>Angular<\/em> work on my codebase show two leagues with nothing in between.<\/p>\n<p>So, for Q4 2026, my recommendation in one line each:<\/p>\n<ul>\n<li><strong>Models:<\/strong> the Anthropic models. <strong>Opus 5.5<\/strong> when quality matters, <strong>Sonnet 5.5<\/strong> when cost matters, <strong>Fable 5.1<\/strong> for planning, design and reviews. <strong>GPT-6.1<\/strong> Sol for small, well-defined tasks and as the cheap second opinion from another provider. <strong>Astra<\/strong> for architecture and planning reviews and computer use tasks.<\/li>\n<li><strong>Harness, on a subscription:<\/strong> whatever you're productive in \u2013 for me that's Claude Code for Fable and Opus, and Codex for Astra and Sol.<\/li>\n<li><strong>Harness, on API prices:<\/strong> <strong>OpenCode<\/strong> or <strong>Pi<\/strong>, with Sonnet 5.5 at medium effort. Same model, half the bill \u2013 medium was not reviewed for quality, so check it on your own tickets.<\/li>\n<li><strong>Orchestrator app<\/strong>, the GUI on top of those coding agents: <a href=\"https:\/\/t3.codes\/\"><strong>T3 Code<\/strong><\/a>, still and by far. Nothing I know of competes with it in functionality and developer experience, and the team behind it really knows what they are doing. Why exactly it is the best option deserves its own post \u2013 that one is planned for right after the cost follow-up.<\/li>\n<\/ul>\n<p>And do your own research: your codebase, your tickets, your reviewers. It costs a few days and a few hundred euros in tokens, and it is the only benchmark that measures what you actually pay for.<\/p>\n<p>Next up: <strong>what agentic coding really costs<\/strong> \u2013 cost per merged pull request, why subscriptions are fair and the API isn't, the four token types and why caching is the reason any of this is affordable, and the context management that cuts the bill. Coming soon.<\/p>\n<p>Thank you for reading \ud83d\ude4f this blog post was written by Alexander Thalhammer. For feedback, remarks or questions, please reach out to me \u2764\ufe0f<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is post 10 of 10 in the series &ldquo;Agentic Engineering&rdquo; Agentic Engineering: Which LLM Is Best for Angular Development? Agentic Engineering: Which App\/Harness Is Best for Angular Development? Agentic Engineering: What Does AI Coding Really Cost? Agentic Engineering: What Do AI Coding Tools Do With Your Code? Agentic Verdict: What&#8217;s the Best Solution for [&hellip;]<\/p>\n","protected":false},"author":21,"featured_media":35148,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_price":"","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":""},"categories":[18],"tags":[],"class_list":["post-35151","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","post_series-agentic-engineering"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.1.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Don&#039;t Trust the Benchmarks. Do Your Own Research. - ANGULARarchitects<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Don&#039;t Trust the Benchmarks. Do Your Own Research. - ANGULARarchitects\" \/>\n<meta property=\"og:description\" content=\"This is post 10 of 10 in the series &ldquo;Agentic Engineering&rdquo; Agentic Engineering: Which LLM Is Best for Angular Development? Agentic Engineering: Which App\/Harness Is Best for Angular Development? Agentic Engineering: What Does AI Coding Really Cost? Agentic Engineering: What Do AI Coding Tools Do With Your Code? Agentic Verdict: What&#8217;s the Best Solution for [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/\" \/>\n<meta property=\"og:site_name\" content=\"ANGULARarchitects\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-29T14:00:26+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-10-29T14:19:51+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-thumbnail-1024x576.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"576\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Alexander Thalhammer\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@LX_T\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Alexander Thalhammer\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"25 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"TechArticle\",\"@id\":\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/\"},\"author\":{\"name\":\"Alexander Thalhammer\",\"@id\":\"https:\/\/www.angulararchitects.io\/en\/#\/schema\/person\/eefb0cd4d115dfd406a02b6dbc760d45\"},\"headline\":\"Don&#8217;t Trust the Benchmarks. Do Your Own Research.\",\"datePublished\":\"2026-10-29T14:00:26+00:00\",\"dateModified\":\"2026-10-29T14:19:51+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/\"},\"wordCount\":4952,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.angulararchitects.io\/en\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-thumbnail.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/\",\"url\":\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/\",\"name\":\"Don't Trust the Benchmarks. Do Your Own Research. - ANGULARarchitects\",\"isPartOf\":{\"@id\":\"https:\/\/www.angulararchitects.io\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-thumbnail.png\",\"datePublished\":\"2026-10-29T14:00:26+00:00\",\"dateModified\":\"2026-10-29T14:19:51+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#primaryimage\",\"url\":\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-thumbnail.png\",\"contentUrl\":\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-thumbnail.png\",\"width\":1672,\"height\":941,\"caption\":\"Four human reviewers examine eight anonymous code submissions around a table beneath the words \u201cReal work. Human judgment.\u201d\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.angulararchitects.io\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Don&#8217;t Trust the Benchmarks. Do Your Own Research.\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.angulararchitects.io\/en\/#website\",\"url\":\"https:\/\/www.angulararchitects.io\/en\/\",\"name\":\"ANGULARarchitects\",\"description\":\"AngularArchitects.io\",\"publisher\":{\"@id\":\"https:\/\/www.angulararchitects.io\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.angulararchitects.io\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.angulararchitects.io\/en\/#organization\",\"name\":\"ANGULARarchitects\",\"alternateName\":\"SOFTWAREarchitects\",\"url\":\"https:\/\/www.angulararchitects.io\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.angulararchitects.io\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2023\/07\/AA-Logo-RGB-horizontal-inside-knowledge-black.svg\",\"contentUrl\":\"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2023\/07\/AA-Logo-RGB-horizontal-inside-knowledge-black.svg\",\"width\":644,\"height\":216,\"caption\":\"ANGULARarchitects\"},\"image\":{\"@id\":\"https:\/\/www.angulararchitects.io\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/github.com\/angular-architects\",\"https:\/\/www.linkedin.com\/company\/angular-architects\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.angulararchitects.io\/en\/#\/schema\/person\/eefb0cd4d115dfd406a02b6dbc760d45\",\"name\":\"Alexander Thalhammer\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.angulararchitects.io\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/23f1b6f9b1ee7d04247b8320851762347d56c76b1537d100d07390d6d919b78d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/23f1b6f9b1ee7d04247b8320851762347d56c76b1537d100d07390d6d919b78d?s=96&d=mm&r=g\",\"caption\":\"Alexander Thalhammer\"},\"sameAs\":[\"https:\/\/x.com\/LX_T\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Don't Trust the Benchmarks. Do Your Own Research. - ANGULARarchitects","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/","og_locale":"en_US","og_type":"article","og_title":"Don't Trust the Benchmarks. Do Your Own Research. - ANGULARarchitects","og_description":"This is post 10 of 10 in the series &ldquo;Agentic Engineering&rdquo; Agentic Engineering: Which LLM Is Best for Angular Development? Agentic Engineering: Which App\/Harness Is Best for Angular Development? Agentic Engineering: What Does AI Coding Really Cost? Agentic Engineering: What Do AI Coding Tools Do With Your Code? Agentic Verdict: What&#8217;s the Best Solution for [&hellip;]","og_url":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/","og_site_name":"ANGULARarchitects","article_published_time":"2026-10-29T14:00:26+00:00","article_modified_time":"2026-10-29T14:19:51+00:00","og_image":[{"width":1024,"height":576,"url":"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-thumbnail-1024x576.png","type":"image\/png"}],"author":"Alexander Thalhammer","twitter_card":"summary_large_image","twitter_creator":"@LX_T","twitter_misc":{"Written by":"Alexander Thalhammer","Est. reading time":"25 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"TechArticle","@id":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#article","isPartOf":{"@id":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/"},"author":{"name":"Alexander Thalhammer","@id":"https:\/\/www.angulararchitects.io\/en\/#\/schema\/person\/eefb0cd4d115dfd406a02b6dbc760d45"},"headline":"Don&#8217;t Trust the Benchmarks. Do Your Own Research.","datePublished":"2026-10-29T14:00:26+00:00","dateModified":"2026-10-29T14:19:51+00:00","mainEntityOfPage":{"@id":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/"},"wordCount":4952,"commentCount":0,"publisher":{"@id":"https:\/\/www.angulararchitects.io\/en\/#organization"},"image":{"@id":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#primaryimage"},"thumbnailUrl":"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-thumbnail.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/","url":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/","name":"Don't Trust the Benchmarks. Do Your Own Research. - ANGULARarchitects","isPartOf":{"@id":"https:\/\/www.angulararchitects.io\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#primaryimage"},"image":{"@id":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#primaryimage"},"thumbnailUrl":"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-thumbnail.png","datePublished":"2026-10-29T14:00:26+00:00","dateModified":"2026-10-29T14:19:51+00:00","breadcrumb":{"@id":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#primaryimage","url":"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-thumbnail.png","contentUrl":"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2026\/10\/ae-bench-thumbnail.png","width":1672,"height":941,"caption":"Four human reviewers examine eight anonymous code submissions around a table beneath the words \u201cReal work. Human judgment.\u201d"},{"@type":"BreadcrumbList","@id":"https:\/\/www.angulararchitects.io\/en\/blog\/ae-human-bench\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.angulararchitects.io\/en\/"},{"@type":"ListItem","position":2,"name":"Don&#8217;t Trust the Benchmarks. Do Your Own Research."}]},{"@type":"WebSite","@id":"https:\/\/www.angulararchitects.io\/en\/#website","url":"https:\/\/www.angulararchitects.io\/en\/","name":"ANGULARarchitects","description":"AngularArchitects.io","publisher":{"@id":"https:\/\/www.angulararchitects.io\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.angulararchitects.io\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.angulararchitects.io\/en\/#organization","name":"ANGULARarchitects","alternateName":"SOFTWAREarchitects","url":"https:\/\/www.angulararchitects.io\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.angulararchitects.io\/en\/#\/schema\/logo\/image\/","url":"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2023\/07\/AA-Logo-RGB-horizontal-inside-knowledge-black.svg","contentUrl":"https:\/\/www.angulararchitects.io\/wp-content\/uploads\/2023\/07\/AA-Logo-RGB-horizontal-inside-knowledge-black.svg","width":644,"height":216,"caption":"ANGULARarchitects"},"image":{"@id":"https:\/\/www.angulararchitects.io\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/github.com\/angular-architects","https:\/\/www.linkedin.com\/company\/angular-architects\/"]},{"@type":"Person","@id":"https:\/\/www.angulararchitects.io\/en\/#\/schema\/person\/eefb0cd4d115dfd406a02b6dbc760d45","name":"Alexander Thalhammer","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.angulararchitects.io\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/23f1b6f9b1ee7d04247b8320851762347d56c76b1537d100d07390d6d919b78d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/23f1b6f9b1ee7d04247b8320851762347d56c76b1537d100d07390d6d919b78d?s=96&d=mm&r=g","caption":"Alexander Thalhammer"},"sameAs":["https:\/\/x.com\/LX_T"]}]}},"_links":{"self":[{"href":"https:\/\/www.angulararchitects.io\/en\/wp-json\/wp\/v2\/posts\/35151","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.angulararchitects.io\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.angulararchitects.io\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.angulararchitects.io\/en\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/www.angulararchitects.io\/en\/wp-json\/wp\/v2\/comments?post=35151"}],"version-history":[{"count":7,"href":"https:\/\/www.angulararchitects.io\/en\/wp-json\/wp\/v2\/posts\/35151\/revisions"}],"predecessor-version":[{"id":35215,"href":"https:\/\/www.angulararchitects.io\/en\/wp-json\/wp\/v2\/posts\/35151\/revisions\/35215"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.angulararchitects.io\/en\/wp-json\/wp\/v2\/media\/35148"}],"wp:attachment":[{"href":"https:\/\/www.angulararchitects.io\/en\/wp-json\/wp\/v2\/media?parent=35151"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.angulararchitects.io\/en\/wp-json\/wp\/v2\/categories?post=35151"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.angulararchitects.io\/en\/wp-json\/wp\/v2\/tags?post=35151"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}