Loop Engineering with Super Mario and Twelve AI Coding Agents

I gave twelve AI coding agents the exact same job: implement a Koopa Troopa — Nintendo's shelled enemy — in the same small Angular-based Super Mario clone, working from the same two tickets, under the same automated agent loop. Then I watched what came back.

This is a backyard test setup, not a leaderboard — n=1, one codebase, one domain, no statistical significance. What I was after isn't a ranking of models but a feel: how an agent loop actually behaves when you swap the model underneath it.

The tasks

The project is a deliberately compact Mario clone: a tile-based level, a hero with jump physics, and one existing enemy — the Gumba — as the reference implementation. Each of the twelve agents got a pristine copy of this baseline and two GitHub issues:

1. koopa-troopa — implement a green Koopa Troopa: stomp it and it hides in its shell, touch the shell and it slides, a sliding shell kills other enemies and Mario (with a short immunity window after the kick).
2. enemy-bounce — when Mario stomps any enemy, he bounces; holding the jump key at that moment gives a higher bounce; and a held key must never trigger repeated jumps (release-latch).

Both tickets are exactly the kind of work I wanted to test: a new feature woven deep into existing behavior (collision system, stomp mechanic, jump physics) rather than bolted on beside it, complexity well beyond CRUD demos, and a nearby example to steer by — the situation you find in large business systems, where new requirements almost always land next to something structurally similar.

Each ticket read like one from sprint planning: a user-facing description, scope and out-of-scope lists, Given/When/Then acceptance criteria, and a verification section ("all tests green, build passes"). The refinement conversation was simulated up front: an LLM posed the clarifying questions a developer would ask, and the answers went back into each ticket as a Design Decisions section. Remediation was sprint-like too: bugs from the first review went into a single QA-style fix ticket, and each agent got exactly one more run. A wrong sprite tile as the only defect went un-ticketed — more on why below.

The harness

Every model ran under its vendor's own official agent — Claude Code, Codex CLI, cursor-agent, Antigravity, Grok CLI — at stock configuration. So this doesn't compare bare models; it compares the products developers actually use, model plus agent plus system prompt and tooling.

Each was wrapped in the same orchestration harness, Sandcastle — essentially a Ralph loop (the agent re-invoked over its own output until the work is done) with four stages: a planner reads the open tickets, an implementer takes each ticket on its own branch and writes tests as it goes, a reviewer passes over the diff, and a merger integrates what survives. Wired to GitHub, a run leaves the same paper trail a human sprint would — issues in, branches and comments out, including a token and wall-clock report per issue.

Each stage ran inside a sandbox. That matters beyond network hygiene: twelve solutions to the same tickets sat in adjacent directories, and each had to be arrived at independently. On the two agents whose transcripts I audited call by call (Grok and Opus 5), that held — no neighbor's repo was ever read.

The results

The numbers are sums across both tasks plus any fix rounds, covering all phases, taken from the reports Sandcastle posted on each GitHub issue. It all happened in July 2026.

Costs are each run's metered usage (fresh input, cache writes/reads, output, all phases and fix rounds) at public API list prices — a normalized what-if, not necessarily what a run costs you: under flat-rate subscriptions the marginal run is free until quota. Fine print: Composer's cache-read rate is assumed (unpublished), Sonnet uses its introductory price, GPT-5.6 the late-July cut.

Every row is one clean, completed run from a pristine baseline. I judged the agents the way a tech lead reviews a pull request: I read every diff, ran each merged test suite locally, and played the level to see the enemy actually move — the only reason the moonwalk below is in this article at all, because no test suite in the field caught it.

The spread: 160× in cost, 19× in wall clock. One number worth decoding: Sonnet metered by far the most raw tokens — 98% of them cheap cached re-reads, its loop re-sending an ever-growing context each step. What costs you isn't turns; it's turns times whatever you let accumulate in context. The cheapest runs were not the worst — though the most expensive one (Fable) was the only flawless one. More thoroughness is not automatically better: for a well-specified feature in a small codebase, the 7-to-25-minute runs were entirely sufficient.

Everyone can build a Koopa. Almost no one can see one.

The headline that would have been unthinkable two years ago: all twelve agents shipped a working Koopa Troopa — the full state machine (walking → shell → sliding → stopped), kick immunity, stomp bounce with release-latch. Four got there on the first attempt, the other eight after their single fix ticket.

Cosmetically it was another story, and the failure was systematic: the sprites. The tasks asked the agents to extract the green Koopa's tiles from a PNG sprite sheet, "analogous to the existing Gumba tiles." This was the hardest sub-problem in the entire setup — and the only one where the fix round didn't reliably help. Eight of the twelve agents picked wrong tiles and were ticketed for it:

Three more (sol, Grok and Opus 5) picked wrong tiles as their only defect; terra even gave its Koopa wings — explicitly out of scope. Opus 5 got both shell tiles right but took the walking frames from the wrong row, so its Koopa patrols the level as a Hammer Bro. Exactly one model chose the right tiles on its own: Fable — also the only agent with no defect of any kind.

The bugs traced back to my spec

The user-visible bugs clustered into three patterns, and checking each against the actual ticket text changed how I think about the failures.

The moonwalk. The most common bug: the Koopa walked to the right while facing left, gliding backwards across the level. Nearly every agent with a fix ticket had it.

But the ticket never mentions facing direction. It says: "Implement analogously to the existing Gumba" — and the Gumba is symmetric, with no mirroring logic whatsoever. The agents faithfully replicated a reference that was silently missing the required property. The moonwalk isn't really a model bug; it's a spec bug.

The shell that wouldn't kick. terra shipped the one genuine state-machine bug: jumping on a stationary shell didn't kick it. The acceptance criteria had it right, but the code-shaped Design Decisions bullet silently omitted the resting-shell case. An agent that trusts that bullet over the prose produces exactly terra's bug — the two halves of my own ticket disagreed by omission.

The counterexample. The enemy-bounce ticket specified its trickiest corner explicitly — a held jump key must never re-trigger a jump — and eleven of twelve agents got it right; only Composer shipped an infinite-jump bug, fixed in its remediation round. Where the spec was explicit, failure was rare and idiosyncratic. Where it was silent or self-contradictory, failure was systematic and near-universal.

That's the pattern: the agents didn't primarily fail at coding; they surfaced the ambiguities in my tickets — twelve times over, in parallel. A human developer bridges spec gaps with genre knowledge, often without noticing the gap. An LLM agent is a merciless literalist.

Code quality: what twelve agents deliver without guidance

Working code is a low bar, so I also measured how the solutions are built — with three instruments covering different angles. An AST tool parsed exactly the functions each loop had touched and recorded the hard numbers: function length and nesting depth. Every merged test suite was re-run locally, with coverage collected over the touched feature files — where a file no test even imports counts as 0%, because coverage tools silently drop such files and would flatter exactly the agents that tested least. And the structural qualities the numbers can't capture went to a blind LLM judge with a fixed 1–5 rubric for two criteria: SLAP and cohesion.

SLAP is the Single Level of Abstraction Principle: within one function, every statement should sit at the same level of abstraction. A function either states the why — orchestrating a behavior as a sequence of well-named steps like stompEnemy() or startKickImmunity() — or it carries the how of exactly one such step: the raw mechanics of a single mechanism. Never both, because the mix is what makes code hard to read: three well-named steps, then twenty lines of inline pixel math, then another step, and the reader has to change altitude mid-function. Cohesion is the sibling criterion: one responsibility per function — and one you can actually name.

The judge saw only anonymized, leak-checked bundles — no model names, no agent-specific paths or branding. And to test the rubric's stability, I re-ran the entire judging pass: every score reproduced within one point.

With no quality guidance, the spread was enormous — from textbook layering to a 110-line collision function with a switch inside a switch, pixel math inline around both. Quality tracked the price tags — as a tendency, not a law: the Claude agents all upper-tier, Opus 5 the only 5/5, the cheapest agents at the bottom — and Grok as the honest exception, top-tier layering for under three dollars. Testing spread just as widely: Sonnet wrote twice as much test code as production code, luna less than half.

Can rules buy the quality back?

I took the weakest agent and added explicit rules — SLAP, helpers must be nameable domain concepts, a hard nesting limit of 2 — enforced by both the implementer and the in-loop reviewer. Deliberately no line-count limit: numeric targets get gamed.

It cuts both ways. Two full points on a blind re-judging — the monolith gone, the hard rule held; recognizing violations is much easier than avoiding them, and the reviewer stage is where that asymmetry pays. But the top tier stayed out of reach, the cost nearly doubled — and the testing got worse: the rules said nothing about tests, so the model quietly reallocated effort from testing to structure. Guardrails steer attention; whatever you leave out of them pays the bill.

Three takeaways that held up

1. The floor has risen further than most teams realize. A stateful game-mechanics feature, delivered working by twelve agent stacks, each within one ordinary bug-fix cycle. The question is no longer whether an agent can implement your ticket — it's what it consumes, what cleanup follows, and how you find the defects.

2. Your spec decides what you get. Every systematic failure traced back to something my tickets didn't say; every explicitly specified behavior was implemented near-universally correctly. A spec steers attention rather than adding it: whatever you leave out, the model decides on its own.

However, this is explicitly not an argument for waterfall. The principles of agility apply here as well: tickets with user stories and acceptance criteria work well for both humans and AI, a conversation with the AI reveals gaps and necessary design decisions, and over several iterations, we gradually move closer to the goal.

3. Quality follows the same rule. With no guidance, twelve agents delivered twelve different implementations — from textbook layering to a 110-line collision monolith. Explicit structure rules moved the weakest model two full points on a blind rubric. Treat quality as part of the spec and state it explicitly.