The bootstrap saga: zombie proxies, silent 404s, and pipefail landmines

Every project on this machine starts with the same line in its run.sh:

porthub bootstrap

One command, one promise: exactly one portless proxy on :443, serving the current routes, with the local CA trusted. The command is 200 lines of bash. This post is about why every one of those lines is a scar. Between April and July, I met every way a local proxy can look alive while being dead, and each one taught the script a new refusal.

Failure one: the fallback proxy and the silent 404

Portless needs sudo to bind :443. Decline the prompt (or never see it, because a script swallowed it) and portless does something helpful: it silently falls back to an unprivileged port like 1355. Now the CLI happily registers routes against the fallback proxy, portless list looks perfect, and the browser, which talks to :443, gets a "No app registered" 404 for everything. Nothing crashed. Nothing logged. Two proxies, each telling its own truth.

This is the worst kind of failure: the degraded state is self- consistent. Bootstrap's answer is blunt: sweep any proxy not on :443 before starting, and if the :443 bind never comes up, find the fallback, kill it so it can't shadow the real thing, and exit non-zero with instructions naming the actual fix (sudo -v && porthub bootstrap). The design position: a fallback that half-works is worse than a failure that explains itself. Portless disagrees; bootstrap overrules it.

Failure two: the zombie inode

July's contribution, and my favourite. The proxy holds routes.json open and watches it for changes. But /tmp/portless occasionally gets regenerated: a CA refresh, a /tmp cleanup. The proxy keeps watching the old, now-deleted inode. New route registrations write to the new file; the proxy never sees them. portless list reads the fresh file and looks correct. pgrep confirms a healthy process on :443. And every single request 404s.

No status check can catch this, because every source of status is telling the truth about a different object. The only test that works is behavioural: take a hostname that is registered, request it through the proxy, and see whether the proxy serves its own "No app registered" page — a string unique to portless's no-route branch, so an upstream app's 404 can't false-positive. If the probe fails, wait four seconds (longer than portless's 3-second route-reload poll, so a proxy that's merely mid-poll isn't executed for lag), probe again, and only then recycle the proxy.

The lesson generalises further than bash: process-existence checks verify the corpse is warm; only a request through the front door verifies the service. Health checks that don't exercise the actual data path are opinions. Every test framework I've built across thirty years converged on the same rule: assert on behaviour, never on process state.

Failure three: the landmines in set -euo pipefail

The saga's recurring gremlin wasn't the proxy at all. It was strict- mode bash. set -euo pipefail is the right default and it is also a minefield when "nothing found" is a normal outcome:

• pgrep exits non-zero when no processes match. On a perfectly clean machine — no proxies at all — the stray-sweep aborted the entire bootstrap before it could start one. Strict mode turned the cleanest possible state into the one state that could never boot.
• grep exits non-zero on an empty routes.json ([]). The staleness probe died silently before its own guard clause could run.
• openssl exits non-zero on a truncated CA file, killing the trust check that existed to fix that CA file.

Each fix is the same idiom: || true and let the following emptiness guard decide. Each one now carries a comment explaining exactly which no-match case it tolerates, because bare || true reads as a shrug and these are anything but. Consuming projects grew the matching discipline: their run.sh fails fast and loudly when bootstrap exits non-zero, instead of sailing on to launch an app whose URLs are dead.

The shape of the final script

Bootstrap is now four verbs in strict order: sweep strays off other ports, ensure a :443 proxy exists, recycle it if a behavioural probe proves it stale, trust the CA (with a keychain-fingerprint check first, and a direct security add-trusted-cert fallback for when /tmp/portless is root-owned and portless can't regenerate it). Then verify, and say so. Idempotent from any starting state — including the states nobody designs for, like "two proxies", "zero proxies", and "one proxy, spiritually absent".

Three months for 200 lines. But the economics are the point: this script runs at the top of every run.sh in every project here, dozens of times a day, in front of a human or an AI agent who will otherwise burn twenty minutes rediscovering the fallback-proxy 404. Infrastructure scripts earn reliability budgets out of proportion to their size — they're the code that runs before anyone is watching. Spend accordingly, comment the scars, and never trust a proxy you haven't asked to serve something.