You're Benchmarking the Wrong Layer

This week a question sat on the Hacker News front page — over a thousand points, four hundred-plus replies: "Has anyone actually replaced Claude / GPT with a local model for daily coding?" The thread split into camps. One side: local is cheaper, private, runs offline — someone ran the math on two RTX 3090s paying for themselves against a cloud subscription in three to four years. The other: local still falls short, and you crawl back to the cloud when it matters.

After four hundred comments, nobody questioned the question itself. Buried in it is an assumption: that what decides the swap is how good the model is — as if the day local scores catch up, the deal is done. I write code through agents every day, and my read is the opposite. Most people are watching the wrong layer.

An agent writing code is never just "a model emits a snippet." It's a loop: read the repo, call a tool, edit a file, run the tests, read the error, edit again, run again. One real task is dozens to hundreds of tool calls, and the model is only the part that gets called over and over inside it. What actually decides whether the whole chain holds — or ties itself in a knot by step 30 — is the layer wrapped around the model. The field calls it the harness (the program that plugs the model into tools, manages context, and handles retries and verification; Claude Code and Codex CLI are harnesses).

The cleanest evidence is hiding in the benchmark's own design. On the latest Terminal-Bench 2.1 — which measures agents doing real terminal work — Fable 5 leads at 80.52, GPT-5.5 at 76.40, Opus 4.8 at 71.91. But the number worth reading isn't the ranking; it's the line in the methodology: every model is run on the same harness, Terminus 2. Why lock the harness? Because the people who built the board know it's a confound big enough to drown the differences between models — leave it free and you're no longer measuring which model is stronger, but whose shell is better. That a serious leaderboard has to nail the harness down is itself the proof that harness and model matter on the same order.

It shows up harder in daily work. Boards like SWE-Bench measure "given a neatly packaged problem, can the model solve it" — intelligence in a vacuum. But your real work isn't solving one problem. It's editing one thing in a 200k-line repo you've never read without breaking three others, holding context across hours. None of those boards test that. And in that HN thread, the people running local models don't complain that it can't write a function — a 30B local model has long been good enough for that. They say it's like a junior engineer you have to hand-hold: you spell out the architecture, break tasks down to atoms, even remember to strip the debug statements yourself. Those are all "stay coherent across fifty steps" failures — failures of the loop, not of raw IQ. And without a good harness under it, the frontier model loses the thread too.

Once you see that layer, local-vs-cloud gets reworded. The harness does all the unglamorous grunt work — landing the model's output into the file accurately, retrying on error, compacting context before it overflows, leaving a roll-back checkpoint after every edit. And that layer is open — the part you can reach in and change yourself. In my own agents, swapping the underlying model often barely moves the output. But the time I rewrote the verification step — make it run its own change and roll back if it fails — the whole system's usability jumped visibly. The model is becoming a utility; the harness is the lever you can actually hold.

So that HN question, I'd answer in a line: don't ask whether a local model is good enough to replace Claude — ask whether your loop is steady enough to run no matter which model you drop in. Crude harness, and even the strongest cloud model just pushes the crash a few steps out. Steady harness, and the day a local model can take over arrives sooner than the boards would tell you.

What's worth swapping was never the model. It's the layer you wrap around it.