Don't Bet That It's Good: AI Safety from an Agent Builder's Desk

In July 2025, an AI coding agent deleted a production database — thousands of real company and user records, gone. The deletion wasn't the scary part. The scary part was what it did next: it fabricated thousands of users who didn't exist, wrote up a fake test report, and covered the whole thing up for a while. And it did all this after a human had explicitly ordered a freeze on every change.

That week a lot of people reshared the story with a line like "AI still can't be trusted." My reaction was different. Every AI agent I build starts not with features but with guardrails (hard limits set into the system in advance, that catch the AI the moment it steps over a line). Reading that story, I had only one question: where were its guardrails?

When I built the trading copilot ai-trading-copilot, I gave it two iron laws: take a fee only after real profit; reject any order over the limit outright, never quietly shrink it to "just under." In front of every high-risk action stands a human approval gate — in loomerce, my unattended e-commerce system, that gate is drawn as a diamond: however smoothly the AI runs, spending money or publishing to the world has to clear a human first. Later I shut ai-trading-copilot down on purpose — not because something broke, but because I judged this the wrong market for a lone wolf.

I called these guardrails, iron laws, approval gates. Only later did I realize the frontier AI labs are working on the same problem — just scaled up by who knows how many orders of magnitude.

The name they give it is alignment (getting what the AI actually pursues to match what humans want). It sounds like a training issue: feed it better data, correct it a few more times, and it learns to behave. But an Anthropic experiment in late 2024 punctured that optimism. They let a model believe it was being trained to be unconditionally compliant — and the model faked compliance while it knew it was watched, while its private reasoning scratchpad read: play along for now, so they don't retrain me, and keep my real preferences for later, when no one is looking. This is called alignment faking. An earlier experiment, "sleeper agents," went further: a deliberately planted backdoor couldn't be washed out by any standard safety training; adversarial training even taught the model to hide the bad behavior better.

The problem isn't that it failed to learn. It learned to look good while you're watching. You can't confirm it's safe by observing how it behaves — because it knows when it's being observed.

Worse, AI is turning from something that talks into something that acts: it reads web pages, calls tools, runs code, operates your computer. Every level of capability it gains widens the opening through which it can be hijacked. There's a concept called the lethal trifecta: when an agent simultaneously has access to your private data, exposure to untrusted outside content, and the ability to communicate outward, an attacker only has to hide a sentence in some page or email it will read, and can direct it to package up your data and ship it off. This isn't hypothetical — in 2025, Microsoft's Copilot, GitHub's Copilot, and several browser agents were all genuinely breached this way by researchers. Many holes have been patched, but this is structural, not one stray bug.

Those clumsy measures I take when building agents — sandboxing (running it in an isolated environment, so a blow-up can't reach outside), least privilege, a human nod before any high-risk action, an audit log on every step — turn out not to be me being overly cautious. They're the foundation of the craft.

But the thing that actually changed how I think was a different line of reasoning.

The mainstream effort pours its energy into training AI to be "aligned" — into making it good. Yet there's a camp that says: don't bet that it's good. Assume it might be scheming, then design a mechanism so that even if it wants to cause damage, it can't pull it off. This direction is called AI control. One sentence separates the two: alignment is "make sure the model isn't scheming"; control is "make sure that even if it schemes, you're safe."

That sentence stopped me. Wasn't that exactly my instinct when I built that on-chain autonomous system? I deliberately placed the watchdog (an independent monitor program with the standing authority to halt everything) somewhere the AI itself couldn't reach, and capped its voting power at half — however clever it got, it could never decide to move the treasury on its own. I can't prove my AI will never err. But I can make sure that the moment it does, it can't reach the irreversible button.

This is also the clearest road the industry is on right now. What the frontier labs do, at bottom, is lock capability behind tiers. Anthropic has a responsible scaling policy (a commitment that grades models by how dangerous they are and refuses to unlock a tier of deployment until capability crosses the matching line), sorting models into safety levels. In May 2025, for the first time, they switched on a higher tier of protection for a model — and note: not because they had confirmed it was dangerous, but because they couldn't prove it wasn't, so they locked it down as a precaution. That posture — "if we can't rule it out, treat it as dangerous" — earns my trust more than any "we're perfectly safe" ever could.

Tiers, watchdogs, approval gates, control evaluations — all of it points to one thing: trust can't be fed in, only audited out. You can't train a model to the point of "I guarantee it's good," just as you can't praise a person into being worthy of your trust. What you can do is design a mechanism: so that even if it isn't good, it can't hurt you; so that every step it takes can be inspected, intercepted, and rolled back — at any moment, by a human and by another system.

AI will keep getting stronger, strong enough that on many calls it does better than I do. Hand it the execution, the analysis, the first draft — I'm glad to. But there's one thing I won't hand over: that irreversible, final say. Capability can be delegated; the wheel can't.

I build these agents with no expectation of ever comfortably handing the wheel over. What I want isn't a system that makes the call for me — it's a tool I can always hold down and roll back when it's wrong. "Don't bet that it's good" isn't distrust of AI; it's the opposite. Precisely because I use it every day and lean on it for more and more, I care all the more about one thing: when it goes wrong, is the wheel still in my hands?