Interrogate the Metal, Not the Notes

A bad time to lose your memory

One of the AI agents I lean on to run my self-hosted setup got reset last week. Not crashed, not confused — reset. A power cut in the office took the machine it was thinking on, and the session simply ended. It was the middle of an ordinary workday — I was awake and heads-down on the very design the agent was carrying out, which is the only reason I was on hand to restart it at all. There’s no failsafe in my setup that brings one of these back; when a session dies it stays dead until I spin up a new one myself and walk it into whatever it inherited. So that’s what I did — and the fresh instance I started had no memory of what the old one had been doing.

Normally that’s a shrug. You re-read your notes and pick up where you left off. Except this time the old instance had been halfway through the most dangerous kind of change you can run across a group of machines: gathering up every credential they use to trust each other, sealing them, and rewiring each box to read from the new arrangement. The goal behind it was a good one — to stop trusting memory with the things that unlock everything, and turn each secret into something you could rebuild from a sealed, versioned record instead of from someone’s recollection. Which is exactly the kind of job you do not want to be interrupted halfway through. The power died in the middle of it. The session ended mid-sentence.

What the new instance inherited was a folder of half-written notes, a log that stopped without a period, and a set of machines that might have been in the state those notes claimed. No continuity of memory, just artifacts. And artifacts lie — not on purpose, but by leaving things out.

Two things told it that it had woken into the middle of something rather than the end of it. There were files in the working tree that had never been committed, and uncommitted work is the fingerprint of a sentence someone is still writing. And the dead session’s last line wasn’t a conclusion at all — it was a warning to itself: don’t start the next step until a specific safety condition is met. You don’t end a finished job by telling yourself not to do the next thing. A log that stops mid-line tells you it stopped. A log that stops on a don’t tells you exactly what it was about to do, and why it hadn’t yet.

What a half-finished migration actually looks like

A migration is a sequence: change one thing, verify it, change the next. Power doesn’t care about your sequence. Cut it mid-stream and you can be left holding almost anything — a credential rotated on one machine but not its partner, so half the fleet trusts the new value and half still trusts the old, with nothing on the surface telling you which is which. A certificate reissued but never deployed, valid on paper and missing wherever it’s actually served. The sharp one: the single credential that unlocks the entire new arrangement, sitting in exactly one place, not yet copied anywhere, with the very next step about to start locking things against it.

The vicious part is that none of it announces itself. A half-applied migration looks identical to a finished one, right up to the moment something reaches for the half that isn’t there.

The new instance got lucky, and it’s worth being honest about that. The cut had landed on the safe side of the worst line — the machine holding the irreplaceable copy never lost power, only a more disposable one did, and the work hadn’t yet crossed the point of no return. The state was frozen, not torn. That’s also why the recovery was quick: a torn state is where you lose hours reconciling which machine believes what, while a frozen one you just confirm, one careful question at a time. The speed was the state’s gift, not any cleverness in the recovery.

The records you inherit are claims, not facts

Here’s the part I keep chewing on. Four different things carried the state forward, and the whole job of coming back was ranking them by how much you can actually trust them.

The plan — the design doc — reads the richest and is the poorest. It tells you what someone meant to do, with nothing in it about how far the work actually got.

The handoff notes are a step better and still soft: a claim about a claim, one session writing down where it thinks it stopped, for a future it never reached.

Version control is firmer. A commit is a thing that genuinely happened — timestamped, immutable, real. But it’s only ground truth about the documents. A line that reads “secrets migrated” proves a sentence got written. It does not prove a secret moved.

And then there’s the live infrastructure: the actual key on the actual disk, the actual file the actual service reads. Everything else is a description of this. Only this is the thing itself.

The discipline comes down to one sentence: never let a description outrank the thing it describes.

Don’t read the checkbox — ask the machine

So the new instance didn’t trust the notes, even the parts marked done. For the keystone credential, the notes said it had been created and recorded its public fingerprint. That’s a claim, and it could be wrong two ways: the file could have been corrupted by the hard shutdown, or the notes could be describing something that no longer existed.

Instead of believing the fingerprint, it went to the file on disk, re-derived the public identity straight from the private material, and checked what it computed against what the dead session had written down. They matched exactly — and that one match did two jobs at once. It proved the file had survived the power loss intact, and it proved the notes were describing the real thing and not a ghost.

That became the pattern for everything the records called done. For each credential: decrypt the stored copy, hash it, hash the live copy the service actually reads, and believe it only if they come out byte-for-byte identical. The sharpest case was a credential where the right move was to never touch the live copy at all. The record would happily have let it re-apply the thing from scratch. Instead it proved the stored version reproduces the live one exactly, hash against hash, and left the running system alone. The verification replaced the action.

I should be honest about what it means that I let an AI near those keys at all: I didn’t trust it. That’s the whole point. I didn’t trust the dead session’s notes, and I didn’t extend any more faith to the new instance’s conclusions — both had to clear the same bar, proving themselves against the live system hash for hash or not counting at all. Verify, never trust, and the “never trust” has to include the thing doing the verifying. The only reason an AI re-deriving my credentials was something I could live with is that nothing it claimed was allowed to matter until the metal agreed.

Where the records left it blind

That’s not the interesting part, though. The interesting part is where the records failed, because that’s the part you can actually fix.

They captured intent, and they captured finished steps. What they never captured was the in-flight one — the exact instruction the dead session was halfway through when the lights went out left no trace at all. The new instance had to reconstruct it by poking at the live machines and reasoning backward.

But the worst gap wasn’t a missing step. It was a missing fact. The single thing the recovery most needed to know — had the keystone credential been copied off its one machine yet? — was written nowhere. The record never said either way. And a silence like that is worse than a wrong answer, because a wrong answer you catch by checking, while a silence you don’t even know to question until it occurs to you to go ask the machines yourself. The record was loudest where the stakes were low and went quiet exactly where they were highest.

Here’s what the silence was hiding. For a window, the entire new arrangement hung on that one un-duplicated file. Losing it right then would have cost almost nothing — it hadn’t sealed anything yet, so you’d make another and start over. The terror is one step later. Once that key has sealed every credential and then is lost before it’s copied, your backups stop saving you and start becoming static — ciphertext bound to a key that exists nowhere in the world anymore, with nothing left that can open it. The only way back is to regenerate every secret from scratch, by hand, re-establishing trust on every machine while everything that depends on those secrets sits broken until its replacement is built. That’s what makes it the keystone and not just another secret: losing it doesn’t destroy your data, it destroys the recoverability of your data. Which is why the one rule that actually mattered was the dull one — copy it the instant it’s born, before it has anything to guard.

And there’s a bitter joke folded into that. The whole point of the migration was that nothing important should ever live in only one place — that’s the entire reason to seal secrets into a versioned, backed-up record instead of leaving them in single copies scattered across the machines. At the moment the power cut, its own keystone was living in exactly one place. The tool built to kill single points of failure had one of its own, for a window, right at its heart. It just hadn’t finished applying its own lesson to itself yet.

The records can fail in the other direction too. There was a setup that everyone’s notes, and everyone’s memory, treated as a live tripwire — a place where one routine command would supposedly wipe out a production system. Traced on the actual machines, the wiring had never connected; the “dangerous” command had never once touched the real thing. The hazard was documented with total confidence and was completely imaginary, and the only way to learn that was to stop reading the warning and watch what the command actually did. A record can invent a danger as confidently as it can stay quiet about a real one.

The checkpoint that saved this whole thing was real design. The dead session had found the single irreversible moment in the job and written a gate in front of it: stop here, duplicate the key, don’t go on. But it was a gate made of prose — a note to a human — and prose stops nothing. The outage didn’t spare the migration because the system refused to cross the line. It spared it because the session happened to be paused at that line, waiting on me to say go. Design picked the right place to stop; luck decided the power cut while it was stopped there. An unenforced checkpoint is only ever as safe as the moment the outage happens to land.

So here’s the lesson I actually walked away with: stop writing your checkpoints as prose for a human to read between the lines of, and start writing them as state a machine can refuse to ignore. “Back up the keystone before you seal anything against it” shouldn’t be a sentence in a document — it should be a precondition the process won’t advance past. “Where did I stop” shouldn’t be a paragraph to interpret — it should be a flat, boring, machine-readable status the next session can read at a glance. Records of what you finished are everywhere. Records of what you’re in the middle of, and what you’ve promised yourself not to do next, are rare — and they’re exactly the ones a power cut makes you wish you’d kept.

The only memory that survived

I want to be straight about where I’m standing, because the honest version beats the heroic one. This wasn’t a careless lunge. The migration was disciplined — every file proven against its own fingerprint before and after, live secrets never overwritten in place, the master key set to be copied to several safe homes the moment it existed. And losing a session is something I’ve put real work into surviving: externalized memory, instructions kept current, documentation that matches what’s actually deployed, a shared record the agents reconcile against, notes written as the work happens. That apparatus is why a dead session is usually a shrug — the next one reads in and picks up clean. It’s also where the confidence came from.

And it’s exactly what fooled me. All of that machinery carries a session’s memory across the gap; none of it was built to carry a half-finished, irreversible operation across one. The deeper irony is that the migration itself was a hunt for single points of failure — secrets scattered across the fleet, every project handling its own, each credential sitting in whatever one place happened to use it, none of them backed up or even on a list, every one a machine fire from gone. I was in the middle of closing those off and walked right past the one hiding in how the work recorded its own progress. The piece I was missing sat in that blind spot: a gate the system itself would refuse to cross until the master was backed up, instead of a sentence in a document reminding me to. I went after every secret I own with that gap invisible to me — somewhere between blissful ignorance and audacity I still can’t quite place. Then the power picked that exact moment to cut. I couldn’t have built a crueler test, and the preparation genuinely helped: the verified steps and the live note are most of why the state was recoverable instead of a smoking hole.

But the part I keep turning over is that the thing which carried the recovery wasn’t new. The migration was already built on one rule — check every secret against the live bytes, never believe a note that says it’s done — and the recovery was that same rule turned around: never accept that a step is finished until the live machine says so itself. Verify, never trust. It’s what the whole place runs on, and it’s the only reason a power cut in the middle of the most dangerous change I run didn’t end as a disaster. A real operations team would have had the machine-enforced gate I didn’t; if you do this for a living and you’re wincing, I’d genuinely like to hear how you’d have built it — that’s most of why I write these down.

And underneath all of it sat the one record that can’t lie. Not the notes, not the plan, not even the commits — the machine itself. Documents describe; infrastructure is. When the two disagree, the metal wins, and the whole skill of coming back from a session that died with its memory is learning to ask the metal directly, in a voice it can’t fudge. You don’t ask whether it was changed. You tell it to show you what it is, and you work out for yourself whether that’s what it should be.

The session before couldn’t tell anyone where it stopped. The machines could. That turned out to be the only memory that survived the outage — and the only one that was ever really worth trusting.

Companion pieces: why a secret you can only restore from memory isn’t really backed up, and how to actually build this, with off-the-shelf tools.