How to Review AI-Written Code Without Trusting the AI (or Yourself)

Somewhere in my workflow there’s a stage I trust more than any other, and it’s the one built on trusting nothing — least of all the agent that just wrote the code.

I’ve written about the whole crew of AI models I run elsewhere. This is one stage of that operation up close: the review. On a job, you don’t pour the concrete because the crew says it’s good — an inspector checks it against the approved plan first, and nobody signs off until it passes. This review is my inspection stage: one part of a longer build lifecycle, with the plan it answers to before it and the sign-off after — an arc that earns its own deep-dive. The change gets built by an agent, and before it goes anywhere it has to survive the review, built on a single rule: a reviewer is worth exactly how unlike the thing it’s reviewing.

That rule isn’t something I read in a book — it’s how I work. I get at a problem by backing away from it, turning it until I can see it from a few different angles, because that’s where I find the better way to organize it, attack it, or get ahead of it. So when I started leaning on AI, that’s what I wanted out of it: something that would vet the work from several angles instead of nodding along from one. And I didn’t take its blind spots on faith. I suspected the models carry the same biases people do — over-weighting what’s recent, what’s close — so I tested for it, and it held. The biases are real, and they’re predictable. Which suits the only job I’ve had since I first ran into what AI gets wrong: find the points where it breaks, and build the tool that snuffs each one out. This is the tool for one of them.

Here’s the problem that rule answers. When you write code — or when an AI writes it for you — you test what you thought to test. Your test suite is the shape of your assumptions: you feed it inputs that look like what you’re building for and check that it does the right thing with them. That’s confirmation, and it only tells you the happy path works. It can’t tell you what your code does with the inputs you didn’t think of, because you didn’t think of them, so you never wrote the test. A builder’s tests confirm; they don’t falsify. And you can’t review your way out of that with more tests from the same head — they just carry the same blind spots at higher volume.

The bug my own tests confirmed for two weeks

A concrete one. My file scanner, file-observer, identifies a file by looking at its contents. One tier of that — the pure-Python fallback it uses when the libmagic C library isn’t installed — scans the front of the file for known signatures. The signature for a PDF is the literal %PDF-, and the code matched it anywhere in the first 8 KB. If %PDF- turned up at byte 864 of a Python test file — one that writes a fake PDF header mid-file — the scanner called the file a PDF.

That’s wrong, but it’s wrong in an interesting way, because matching anywhere is exactly right for a different question. If you’re asking “is there a PDF buried somewhere in this file?” — polyglot detection, where one file is deliberately valid as two formats at once — you want to find the marker wherever it hides. The bug was born when a new “what type is this?” check reused the matcher from the “is one buried in here?” check. A behavior that was correct in one place leaked into a place where it wasn’t. And it sat there for two and a half weeks, through every release and every leg of my review, because every test I’d written fed the scanner real PDFs — marker at the front — and checked that it said “PDF.” Not one test fed it a non-PDF that happened to contain the marker. My tests asked “does a real PDF get recognized?” They never asked “what does this do to a file that isn’t a PDF but contains our marker?”

What caught it was a clean-room reimplementation — a second version of the same capability, built from scratch against a strict justify-every-behavior bar. Building the PDF rule from zero, it hit the same fork I had — anchor the marker to the front, or match anywhere? — and it had no reason to match anywhere for a type judgment, so it couldn’t justify it, so it flagged it. It didn’t know what my code did. That’s the whole point. It couldn’t inherit the assumption I’d stopped looking at, so it audited it for free.

Four reviewers, chosen to fail differently

That clean-room twin is one of four reviewers a change has to clear, and they’re picked to break differently from each other:

• A swarm of in-house agents, each taking a different angle on the change.
• A different model family entirely — not another instance of my usual model, a genuinely different mind, because copies of one model share its blind spots.
• A deterministic sweep against real data — no opinions, just the code run over a real corpus to see what it actually does.
• The outside PR bots, when it opens as a pull request.

Most of this isn’t mine, and that’s the point — a review gauntlet isn’t a clever invention, it’s an assembly. Opening a change as a pull request and letting bots pick at it is standard practice; the in-house swarm is a red-team skill someone else built that I just aim at my own code. What’s actually mine is duller than a custom review rig: choose checkers that fail differently from each other and from me, and believe none of them until I’ve reproduced it.

Each leg asks a different question. The builder asks “does it do what I intended?” A security-minded model asks “where do I break this?” The deterministic sweep asks “does it do the same thing twice?” The overlap between those questions is small, and the gaps between them are where the bugs live.

Two quick examples, because the pattern matters more than any one catch. The different-model leg — Gemini, in my case — once found a determinism bug my in-house swarm had walked right past: when the scanner couldn’t read a file’s timestamp, it stamped the record with the current wall-clock time, and that timestamp fed a checksum that’s supposed to be identical across runs. So scanning the same unreadable file twice produced two different results, in a tool whose entire promise is that it doesn’t. My swarm had been hardening the thing against crashes; it wasn’t asking about determinism. A different model, asking a different question, found the class I wasn’t hunting.

And the bots catch my fixes, not just my code. I once patched a crash by coercing a value to a string with str(). A PR bot — Codex — pointed out that str() on that kind of object returns something like <object at 0x7f3a…> — a memory address, which changes every run. I’d traded a crash for another determinism break and been perfectly satisfied with it. The reviewer caught the fix I’d already signed off in my own head.

The honest ledger

Now the part the people selling you on AI code review leave out. A reviewer that doesn’t share your blind spots buys you coverage, not correctness — and the gap between those two is where it costs you.

The scariest findings are often the wrong ones. That same Gemini audit that found the real determinism bug also raised two louder alarms — a classic XML-expansion bomb and a zip bomb (booby-trapped files built to balloon into gigabytes the moment something opens them), both flagged high severity. Both were wrong. I built an actual repro for each, and the scanner already refused them: one hits a built-in expansion limit, the other rejects oversize entries and fails closed. The two findings with the scariest labels were the two that evaporated the moment I tried to reproduce them. Act on the severity instead of the repro and I’d have “fixed” two things that weren’t broken.

Even a correct finding can carry a wrong fix. On that same PDF bug, a bot correctly noticed my new 256-byte header window was tighter than a 1,024-byte limit the rest of the code used — a real inconsistency. Its suggested fix was to widen mine to 1,024. But the stray marker that started the whole mess sat at byte 864 — past my 256-byte window but inside 1,024 — so taking the advice would have walked the original bug right back in. A true observation wearing a fix that reintroduces the thing it observed.

And the bots are noisy. They overstate severity, they confidently critique code they were never shown, and one of them, going by my own rough count, reposts findings I’ve already fixed something like a third of the time. Which is why the rule under all of it is non-negotiable: a finding doesn’t count until I’ve reproduced it against the real code. Not “three reviewers agree” — agreement is cheap, and here it was wrong twice in one audit. Confirmation is a repro I built, not a tally of who voted. The whole apparatus is only worth running because I pay that triage tax every single time. Skip the tax and four reviewers chosen to fail differently just become four confident-noise generators.

It also gets less useful the better the code already is. The first time my in-house red-team ran it found six real issues; the next time it ran, exactly one — the first pass had closed the rest. On a small, well-reviewed change, running the full gauntlet can cost more attention than it catches. That’s not a failure; low yield is the upstream discipline working. But it’s a real cost, and pretending otherwise is how you end up with process for its own sake.

And one cost has nothing to do with the findings at all. Running this many machines means handing the platform that many chances to say no. Fan out a swarm of agents, or batch the work too hard, and you can trip a provider’s rate or abuse limits and stall — or lose — a session mid-run. The AI services are growing fast and throttling unpredictably; lean on them hard enough and a whole leg of your review just stops. I’ve built in ways to soften the blow, but caught at the wrong moment it costs real money getting a fresh session back up to speed. More reviewers means more machines, and more machines means more surface for one of them to go dark.

So here’s the honest version, which I find more convincing than the highlight reel anyway. A reviewer that doesn’t share your blind spots will find things you structurally cannot — that’s real, and it’s worth a lot. It will also hand you confident nonsense, scary-labeled non-bugs, and correct findings with broken fixes, and the only thing that sorts the signal from the noise is reproducing every claim against the running code. Reviewers unlike me, and nothing they flag believed until I’ve reproduced it. You need both halves. Reviewers who don’t share my blind spots but check nothing are just a louder argument; checking everything with no one unlike me to do the flagging is just me, agreeing with myself.

One last honest note, since the whole thing runs on honesty: I’m an enthusiast. I self-host, I read what I can, and I build with whatever I manage to figure out. In my actual life I can go months without crossing paths with anyone who knows what Python or Git or markdown is, let alone front matter — so this is the best I’ve worked out on my own, and it keeps changing as I learn and bolt on new tools. I have no doubt someone who does this for a living could read it and find holes I can’t see from the inside. Which, given everything above, is the one review I’m still missing and would want most. I look forward to the day someone hands it to me.