Open a file and you get what it wants to show you — the words, the layout, the finished thing. What you don’t get is the other layer: the name of whoever authored it, the software that built it, the timestamps, the quiet trail of who touched it and with what. Almost every file you own carries some of that, tucked into bytes you never see when you double-click. Almost nobody looks.
I built a tool that looks. Point file-observer at a directory and it makes a single read-only pass and hands back one JSON manifest of what’s actually in there — every file’s type, structure, and the metadata hiding under the surface. It reads everything and changes nothing. (The name sounds like a file watcher; it isn’t. A watcher stays resident and reacts to changes. This runs once, on demand, and exits.)
The tool is simple to describe. The part worth writing down is the discipline of reading that hidden layer honestly — because the moment a tool starts editorializing about what it digs up, you can’t trust a word it says.
A file is a performance; the bytes are the memory
A rendered document shows you what it wants you to read. The file underneath remembers who touched it, with what, and when, in a layer the performance never surfaces. Every author field is a small signature nobody meant to sign; every “producer” string a fingerprint of a tool and a moment. People assume those traces evaporate. They don’t — they sit in the bytes, patient, until something is built to read that deep.
So the real job was never “read the metadata.” Reading it is easy. The job is reporting what’s down there without lying about it, and that turned out to be the whole design.
Report the news, don’t write the editorial
The rule I kept coming back to: observe, don’t interpret. The scanner reports what it can see and stops there. It doesn’t decide whether a file is dangerous, whether a document is “important,” or what you should do next. It records; the consumer interprets.
That sounds like a limitation. It’s the opposite — it’s the thing that makes the output trustworthy. The moment an observation tool starts editorializing (“this file looks risky,” “this is probably spam”), you can no longer tell its facts from its opinions, and you inherit its judgment whether or not it fits your situation. Keep the two separate and the same manifest serves an audit, a classifier, and a security review without lying to any of them.
Four habits fall out of that one rule, and they’re the part I’d take to any data tool I build next.
null means “not observed,” not “not present”
This is the single most important honesty in the whole design, and the easiest to get wrong. The scanner reads within declared bounds — a capped preview, a bounded byte window, a per-format extraction budget. When a field comes back empty, it means “I didn’t see this within the limits I was given,” not “this isn’t in the file.”
Most tools blur that line, and it quietly poisons everything downstream: a null that secretly means “didn’t look” gets read as “isn’t there,” and someone makes a decision on an absence that was never checked. Drawing the line in bright paint — null is “not observed within bounds,” full stop — is what lets a consumer trust a present value and know exactly what a missing one does and doesn’t tell them.
Every signal carries its receipts
Every layered signal in the manifest — not only the computed ones, the directly-read ones too — carries a provenance entry: which layer it came from (read straight off the bytes, or derived from other signals), which method produced it, what triggered it. A MIME type tagged raw says “this came straight from libmagic”; a mismatch flag tagged derived says “I computed this by comparing two other signals.” Nothing is a bare assertion you take on faith. The same idea extends to the observation vectors: each one’s rules are hashed into an identity digest, so the day you change what a vector means, its identity moves with it — you can’t quietly redefine a signal and keep its old name. When you’re debugging why a pipeline did something six steps later, “this field, from this layer, by this method, on this trigger” is the difference between a fix and a guess.
Safety flags are observations, not verdicts
The scanner can flag that a PDF contains JavaScript, or that an Office file has macros. Those are structural facts — “this was observed” — not a score and not an accusation. The tool never quarantines, never rates, never says “dangerous.” It surfaces the structure and leaves the threat model to you, because your threat model is the only one that knows whether a macro in that file is a red flag or completely expected.
Determinism, or it isn’t evidence
Same files, same scan context, same manifest. The guarantee is precise. It doesn’t claim byte-identical output on every machine by magic; instead the platform, the logic version, and the dependency versions are recorded as part of the context, so any legitimate variance is explained by something in the manifest rather than floating free. The volatile bits — a scan id, a timestamp — are deliberately left out of the manifest’s checksum, so two scans of an unchanged tree match. It even holds across parallelism: run it single-threaded or across many workers and the output is byte-identical. I gate every release against detection drift over an ~18,800-file corpus, so “deterministic” is a tested promise rather than an aspiration. An observation you can’t reproduce is just a rumor — reproducibility is what lets you diff two scans and trust that a difference means the files changed, not the weather.
What it actually looks like
flowchart TB
subgraph IN [A messy directory]
direction TB
f1["report.pdf"]
f2["data.xlsx"]
f3["mail.msg"]
f4["...thousands more"]
end
obs["file-observer - one read-only pass"]
man[("one deterministic JSON manifest")]
subgraph OUT [Many consumers downstream]
direction TB
c1["ingest / OCR pipeline"]
c2["classifier"]
c3["audit"]
c4["security review"]
end
IN -->|"reads everything, changes nothing"| obs
obs --> man
man --> OUT
Discipline is abstract; the output isn’t. Pointed at a project, the human-readable summary is a few honest lines — counts of text vs binary, how many files got specialist metadata, how many came back clean vs degraded, which observation vectors matched. The real artifact is the JSON: one object per file, with its checksum, detected type, structural signals, any safety flags, and the provenance trail behind every layered signal — the whole manifest sealed with a checksum (and an optional HMAC signature) so you can prove it wasn’t tampered with between scan and use.
Here’s a single trimmed file object — a Markdown file whose content libmagic reads as plain text even though the .md extension implies Markdown:
{
"path": "Documentation-Review-2026-04-02.md",
"mime_type": "text/plain",
"checksum_sha256": "5c6a7abe4037aff0…",
"mime_analysis": { "detected_mime": "text/plain", "extension_mime": "text/markdown", "matches_extension": false },
"safety_flags": [],
"signal_provenance": {
"mime_type": { "layer": "raw", "method": "detect_mime", "trigger": "libmagic" },
"mime_analysis.matches_extension": { "layer": "derived", "method": "analyze_mime", "trigger": "mismatch", "inputs": ["mime_type"] }
}
}
Notice what it does with the mismatch: it reports matches_extension: false rather than quietly “correcting” the type to match the extension. safety_flags is an empty list — an absence of observed structure, not a clean bill of health. And every signal says where it came from: raw straight off the bytes, derived computed from other signals.
What it deliberately doesn’t do
Being honest about absence applies to the tool’s scope, too. file-observer is an observation layer, not a parser or a full-content extractor. It reads envelopes and structural signals within a budget — page counts, headers, sheet names, titles — not complete document bodies. When a file defeats it, that failure becomes an error record inside the manifest rather than a crash — the scan always finishes and always emits, and a file it couldn’t read is reported as exactly that, in-band, like any other observation. (The robustness bar is “never crash,” proven on a ~50,000-file shakedown that has to complete without a single fatal error; that’s a separate promise from extraction being perfect, and worth not conflating.) The other jobs — OCR, embedding, classification, recovering a corrupt or encrypted file — are downstream, out of scope on purpose, and the manifest says plainly where it stopped looking rather than pretending it saw everything. A tool that’s clear about its edges is easier to build on than one that blurs them to look more capable.
The bugs you only see when you’re honest
Because the scanner is deterministic and runs on real, messy directories, it surfaces its own failures the way it surfaces everything else — visibly, in the manifest. Two that stuck with me:
- A date field on one email format (Outlook
.msg) shipped empty on essentially every real file across three minor releases. Nothing crashed; the field was simply, quietlynull. The synthetic tests passed — it only became obvious when a scan of a real corpus showed 100% null where there should have been dates. The honest null made a silent bug loud. The fix was reading the timestamp from the right place in the format: a properties stream, not the substream the code was checking. - A path-detection vector once reported about 32,000 “file paths” in a self-scan — and 98.6% of them were fragments of URLs buried in JSON (OAuth scope strings), not paths at all. The determinism made the inflated count reproducible run to run; the provenance made it traceable to the exact over-matching rule. A tightened rule dropped it to the few hundred that were real.
Neither is a heroic debugging story. That’s the point. A tool built to report the news honestly will, sooner or later, report the news about itself — worth more than one that hides its own gaps behind confident output.
The name nobody typed
The first time the tool showed me something I didn’t already know, it wasn’t a bug. I pointed it at a big pile of ordinary business paperwork — specs, forms, plan sets, the documents that move through a company by the thousand and that nobody reads twice — and turned on the vector that does one dumb, relentless thing: read the author and producer fields, normalize the names, and count them.
One name kept coming up. Not in the text of anything — underneath it. Stamped into file after file by whatever software the person had used, years of small, unwitting authorship fossilized in the header bytes. They were the most-present person in a corpus that, on its face, mentions them nowhere.
That’s the performance-versus-memory gap made concrete, and it’s the whole reason “observe, don’t interpret” is worth the trouble. The scanner won’t tell you that name mattered, or why — it isn’t clever and it doesn’t care. It only says: here, and here, and here. It points at the layer beneath sight and reports what’s actually there. And in doing only that — refusing to judge, choosing only to witness — it becomes the thing that keeps a person visible: a small, accidental immortality for someone you’d otherwise never have found. There’s a name like that in almost every corpus.
And it isn’t only other people’s files. The next document you send out carries your name, your tools, your timestamps the same way — a small signature you never knew you were leaving. The tool just makes the leaving legible.
Try it
It’s open source (AGPL, with a commercial option) and a pip install away:
pip install file-observer
fo ./your-project --specialists
pypi.org/project/file-observer · github.com/russalo/file-observer
Aside. I didn’t set out to build a file scanner — I needed one. PKP, my personal knowledge pipeline, had an ingester that needed to know what it was about to swallow, and make sure it didn’t swallow anything unsafe; Project Sentinel needed to work out the schema buried across a heap of chat logs. file-observer started as the small thing that answered those: a tool built to serve other tools. Because it was feeding other programs rather than a person, it couldn’t afford to editorialize; those programs needed plain facts they could act on. That’s where “report the news” came from.
There was a second pull, too. I was genuinely curious what metadata is even out there — what schemes exist, what different programs quietly stash away and rely on. Building the scanner turned out to be the best way to find out, and I came away knowing far more than I went in for: front-matter conventions, hierarchical metadata topologies, formats I’d never have cracked open by hand. The need got it started; curiosity is what kept me adding to it, until the side-project-for-my-projects had its own scope and heading, well past what I first needed.
(Built solo, for the record — with a lot of AI pair-programming, which is a post for another day.)
# comments (0)
no comments yet — be the first.