The bootstrap saga: zombie proxies, silent 404s, and pipefail landmines

Every project on this machine starts with the same line in its run.sh:

porthub bootstrap

One command, one promise: exactly one portless proxy on :443, serving the current routes, with the local CA trusted. The command is 200 lines of bash. This post is about why every one of those lines is a scar — because between April and July, I met every way a local proxy can look alive while being dead, and each one taught the script a new refusal.

Failure one: the fallback proxy and the silent 404

Portless needs sudo to bind :443. Decline the prompt — or never see it, because a script swallowed it — and portless does something helpful: it silently falls back to an unprivileged port like 1355. Now the CLI happily registers routes against the fallback proxy, portless list looks perfect, and the browser — which talks to :443 — gets a "No app registered" 404 for everything. Nothing crashed. Nothing logged. Two proxies, each telling its own truth.

This is the worst kind of failure: the degraded state is self- consistent. Bootstrap's answer is blunt — sweep any proxy not on :443 before starting, and if the :443 bind never comes up, find the fallback, kill it so it can't shadow the real thing, and exit non-zero with instructions naming the actual fix (sudo -v && porthub bootstrap). The design position: a fallback that half-works is worse than a failure that explains itself. Portless disagrees; bootstrap overrules it.

Failure two: the zombie inode

July's contribution, and my favourite. The proxy holds routes.json open and watches it for changes. But /tmp/portless occasionally gets regenerated — a CA refresh, a /tmp cleanup. The proxy keeps watching the old, now-deleted inode. New route registrations write to the new file; the proxy never sees them. portless list reads the fresh file and looks correct. pgrep confirms a healthy process on :443. And every single request 404s.

No status check can catch this, because every source of status is telling the truth about a different object. The only test that works is behavioural: take a hostname that is registered, request it through the proxy, and see whether the proxy serves its own "No app registered" page — a string unique to portless's no-route branch, so an upstream app's 404 can't false-positive. If the probe fails, wait four seconds — longer than portless's 3-second route-reload poll, so a proxy that's merely mid-poll isn't executed for lag — probe again, and only then recycle the proxy.

The lesson generalises further than bash: process-existence checks verify the corpse is warm; only a request through the front door verifies the service. Health checks that don't exercise the actual data path are opinions.

Failure three: the landmines in set -euo pipefail

The saga's recurring gremlin wasn't the proxy at all — it was strict- mode bash. set -euo pipefail is the right default and it is also a minefield when "nothing found" is a normal outcome:

• pgrep exits non-zero when no processes match. On a perfectly clean machine — no proxies at all — the stray-sweep aborted the entire bootstrap before it could start one. Strict mode turned the cleanest possible state into the one state that could never boot.
• grep exits non-zero on an empty routes.json ([]). The staleness probe died silently before its own guard clause could run.
• openssl exits non-zero on a truncated CA file, killing the trust check that existed to fix that CA file.

Each fix is the same idiom — || true and let the following emptiness guard decide — and each one now carries a comment explaining exactly which no-match case it tolerates, because bare || true reads as a shrug and these are anything but. Consuming projects grew the matching discipline: their run.sh fails fast and loudly when bootstrap exits non-zero, instead of sailing on to launch an app whose URLs are dead.

The shape of the final script

Bootstrap is now four verbs in strict order: sweep strays off other ports, ensure a :443 proxy exists, recycle it if a behavioural probe proves it stale, trust the CA (with a keychain-fingerprint check first, and a direct security add-trusted-cert fallback for when /tmp/portless is root-owned and portless can't regenerate it). Then verify, and say so. Idempotent from any starting state — including the states nobody designs for, like "two proxies", "zero proxies", and "one proxy, spiritually absent".

Three months for 200 lines. But the economics are the point: this script runs at the top of every run.sh in every project here, dozens of times a day, in front of a human or an AI agent who will otherwise burn twenty minutes rediscovering the fallback-proxy 404. Infrastructure scripts earn reliability budgets out of proportion to their size — they're the code that runs before anyone is watching. Spend accordingly, comment the scars, and never trust a proxy you haven't asked to serve something.