The Caddy Bug That Blamed Let's Encrypt for DigitalOcean's Mistake

Here’s a debugging lesson worth more than the Friday evening it cost me, so I’ll put it first: an error message tells you where a program gave up — not always where it went wrong.

Mine pointed straight at certificate issuance. The real bug was a DNS plugin failing to clean up after itself, two layers from where the error seemed to live. The misdirection is the interesting part — and the fix is one line.

What it looked like

Caddy was issuing a wildcard certificate — *.test.example.com — using a DNS-01 challenge against DigitalOcean.

(Why DNS-01 and not the simpler HTTP-01? The host isn’t publicly reachable — it lives behind a VPN, on CGNAT addresses Let’s Encrypt can’t connect to — so HTTP-01, which needs an inbound hit from the CA, was off the table. For a wildcard on a non-public host, DNS-01 is the only path. Which is how I ended up leaning on the DigitalOcean DNS plugin in the first place.)

The cert would get tantalizingly close to landing, then the whole thing dropped into a retry loop. In the journal:

A string-to-integer conversion, failing on an empty string, in the middle of getting a TLS certificate. Nothing in that error mentions certificates, DNS, or DigitalOcean — so the instinct is to assume the cert flow is broken and start digging into ACME, Let’s Encrypt rate limits, DNS propagation. All dead ends.

What was actually happening

Here’s the whole dance in plain terms. To prove you own a domain, Let’s Encrypt has Caddy write a temporary token at a specific DNS address — like jotting a one-time password on a sticky note and taping it to the door. Let’s Encrypt comes by, reads the note, confirms you own the place, hands over the certificate, and leaves. Caddy’s job is three steps: (a) put the note up, (b) hand the certificate to your web server, (c) take the note back down afterward.

Step (c) was broken. The plugin put the note up fine. The certificate was issued fine. But when Caddy went to take the note down, the plugin crashed on a piece of information it expected and didn’t have. And because the cleanup crashed, Caddy treated the entire transaction as failed — and started over from scratch. Again. And again. Each retry taped another sticky note to the door, and none of them ever came down.

From the outside, all you could see was: “my new site has no HTTPS, and somehow my DNS is filling up with junk.” The certificate had actually been issued several times over. Caddy just never noticed it was succeeding.

The root cause

Underneath the analogy: the bug was in the tagged release of the caddy-dns/digitalocean plugin. On the delete path — taking the note down — it handed an empty string to strconv.Atoi, Go’s “turn this text into a number.” Most likely a TTL or record-ID field that DigitalOcean’s API returns in a different shape on the delete call than the plugin expected. Empty string in, Atoi errors, cleanup fails, and the whole cascade kicks off.

And here’s the genuinely frustrating part: the log message blamed something it had nothing to do with. strconv.Atoi: parsing "": invalid syntax, buried in the cleanup step, means nothing unless you already know how the plugin tracks which note belongs to which transaction. It read like a certificate problem. The bug was in record bookkeeping.

The fix (one line)

Upstream master looked like it carried the fix — but it had never been cut into a tagged release. And the latest tag is exactly what the plugin’s README tells you to build from, so it’s what everyone copy-pastes — with no guarantee that tag wasn’t still shipping the same bug.

So the fix is to build Caddy against master instead:

And I want to be honest: this was not the safe choice, and it wasn’t a sure thing. Pinning @master means running an untagged, in-development branch — no guarantee it was actually free of the bug, and along with the likely fix you inherit whatever else has landed since the last release, minus the (admittedly thin) guarantees a tag is supposed to carry. Nobody could promise me @master would work, any more than they could promise the next tagged release wouldn’t have the exact same problem. It was a gamble. It was also the only move that even might get me a working wildcard cert, so I took it. One line, and the retry loop stopped. Home run — for now.

The alternatives I weighed

When a dependency breaks, the loudest temptation is to tear out the thing that depends on it. The whole reason I was on wildcard DNS-01 was one elegant property: add a hostname under *.test, get HTTPS automatically. Every escape route meant giving that up:

• Drop the wildcard, do per-subdomain HTTP-01. Means making each hostname publicly reachable — a regression on the private, internal-only access pattern — or playing split-horizon DNS games. Every new project becomes a cert-orchestration chore.
• Self-signed wildcard / private CA. Wildcards keep working, but now every device that hits *.test needs the CA installed. I’d be trading an automation problem for a fleet-trust problem.
• No TLS on the internal network. Plain HTTP behind the mesh. Costs you every secure-context browser feature — clipboard, service workers, geolocation — anything that requires HTTPS.
• One named cert via HTTP-01 with a one-time public DNS flip. Works for exactly one hostname and doesn’t generalize.

Every one of those redesigns the architecture to route around a bug. So before any of them, the smaller question: what if I just fix the broken thing? The broken thing was the plugin, not the approach — and there the choice was narrow:

• Stay on the latest tag — keep building against a known-broken release. No.
• Bisect to the exact fix commit and pin that SHA, then PR or file it upstream. The correct answer if I were shipping to customers: precise, reproducible, gives back to the project. For a homelab, a lot of ceremony to reach the same working binary.
• Pin @master and document why. Good enough for a personal setup: it works, and the reason is written down.

I took the last one. It kept the architecture fully intact and fixed the actual broken thing — the @master swap landed in about fifteen seconds. The right amount of rigor depends on who’s downstream of you; for a one-person homelab, “it works and I know exactly why” clears the bar.

The lesson

Four takeaways:

1. An error names the layer that gave up, not always the layer that broke. The message lived in strconv.Atoi; the bug lived in API-response handling; the symptom showed up as failed certificate issuance. Read the literal error before you accept the story wrapped around it.
2. The latest release tag is not a promise that it’s fixed. A tag is just where someone last cut a release. If you’re chasing a known bug, check whether master already fixes it before you lose a night — the patch may be sitting there, untagged. Just know the trade: master hands you the fix and everything else unreleased along with it. Sometimes that’s the right gamble. Make it with your eyes open.
3. A successful operation can still report failure if its cleanup fails. Watch for flows that fold “did the real work succeed?” and “did teardown succeed?” into a single verdict. They are not the same question.
4. Fix the broken dependency before you redesign around it. A bug in one plugin is not a reason to tear out a good architecture. The expensive mistake here would have been abandoning wildcard DNS-01 — and the “add a hostname, get HTTPS for free” automation it buys — to dodge a five-character bug. Fix the small broken thing; keep the design.

Footnote: it came back

Honesty, because the clean version — risky fix, instant home run, done forever — isn’t the true one. On a later renewal, well after pinning @master, the same strconv.Atoi: parsing "": invalid syntax reappeared on the same cleanup path. This time the cert was obtained successfully and only the temporary record was orphaned, left behind at DigitalOcean instead of deleted. So either there’s an upstream regression, or the original fix missed a second code path.

The practical fallout: orphaned _acme-challenge TXT records quietly pile up at DigitalOcean across renewals until someone clears them by hand. Less catastrophic than a retry loop — but it means this one isn’t truly closed, just downgraded from “outage” to “litter.” I’ll update this if I find the second cause.

Aside — the war story. The first round cost me a Friday evening, almost all of it spent in the wrong layer: staring at ACME logs and DNS propagation because that’s what the error had dressed itself up as. The fix itself took seconds, once I stopped trusting the error’s choice of vocabulary.