A deploy that rolls itself back

Before any of the safety mechanics make sense, one decision has to be in place: the configuration that runs on the server is not authored on the server. It lives in the repository, versioned like code, and the deploy installs it onto the box. The live server is a copy of a source of truth, never the source itself. This sounds pedantic until the day someone edits the live config by hand to fix something at 3am, and now the running state and the repo disagree, and nobody knows which is correct.

Making the repo authoritative buys two things at once. The obvious one is history: every change to how the server is configured is a commit, reviewable and revertable like any other. The subtle one is that it makes rollback meaningful. You can only cleanly roll back to a known-good state if a known-good state is written down somewhere you trust — and 'the way the box happened to be configured before I touched it' is not that place. The repo is.

The precondition for any automatic rollback is that you captured the thing you're about to overwrite before you overwrote it. So the deploy's first move, before it installs the new server config, is to copy the currently-live config to a backup. This is cheap, unglamorous, and load-bearing: the undo is only possible because a known-good version was set aside a half-second before the risky change. Skip this step and 'roll back' becomes 'try to reconstruct the previous state from memory,' which is not a plan, it's a prayer.

# snapshot the live config before installing the new one
cp -a /etc/nginx/sites-available/site.conf \
      /etc/nginx/sites-available/site.conf.bak

# then install the repo's version (source of truth)
install -m 0644 deployment/nginx/site.conf \
        /etc/nginx/sites-available/site.conf

The snapshot is the same habit that governs every command I let near production: before it changes anything, it records the exact state it's about to overwrite. Applying and snapshotting are one atomic move, not two things I have to remember to do in the right order under pressure. The rollback path isn't written after the fact when something's already on fire — it's written into the forward path, so the undo exists the instant the change does.

The next move is to ask the server to check the new config without actually running it. Most serious web servers have this — a dry validation that parses the config and tells you whether it's syntactically sane and internally consistent, without taking the risk of reloading it into the live process. For nginx that's a single command, and it's the hinge the whole pattern turns on: it's a chance to catch a broken change while the old, working one is still serving traffic.

# validate the new config WITHOUT loading it into the live process
if nginx -t; then
    systemctl reload nginx          # only reload once it passes
else
    cp -a site.conf.bak site.conf   # restore the snapshot
    nginx -t && systemctl reload nginx
    exit 1                          # fail loud — declined to ship it
fi

If the validation fails, nothing has gone live yet, and the rollback is trivial because the old config was never replaced in the running server. Only if the validation passes does the deploy actually reload the server to pick up the new config. The ordering is the entire point: validate, then commit — never commit and then hope. A change that would have taken the site down instead fails a check and gets caught, at the one moment when catching it costs nothing.

The cheapest place to catch a broken config is in a validation step while the old one is still serving traffic. After the reload, it's an outage. Before it, it's a log line.

If the validation fails, the deploy does not stop and wait for me to notice. It restores the snapshot it took, re-validates that the restored config is good, reloads the server back onto it, and exits loudly with a failure. The site never left the last-known-good state, and the deploy's exit tells me plainly that it declined to ship a broken change. The recovery happened at machine speed, in the same script, without depending on me being awake and clear-headed.

This is the difference between a deploy that's safe and one that merely usually works. A deploy that usually works is fine right up until the night it doesn't and you're the rollback mechanism, half-asleep, trying to remember what 'good' looked like. A deploy that rolls itself back has already answered that question in advance, in code, tested along the same path as the forward deploy. It's the same reason I put a deliberate human gate before the irreversible part of the pipeline and let the machine run freely everywhere else: the safety lives in the structure, not in my attention at the moment it's needed most.

Config validation catches broken syntax, but it can't catch a change that's valid and still wrong — the reload succeeds, and the service behind it is somehow unhealthy anyway. So the last step is to actually exercise the running system: after the reload and the service restarts, the deploy hits the real health endpoints and waits, because reality is the only thing that actually answers the question 'is it up?'

# poll the real endpoint with backoff — a service can take
# well over 5s to bind its port after a restart
for i in $(seq 1 20); do
  curl -fs http://localhost:8000/admin/ >/dev/null 2>&1 && break
  echo "  waiting for the app ($i/20)..."; sleep 3
done
curl -fs http://localhost:8000/admin/ >/dev/null 2>&1 \
  || { echo "not healthy after 60s"; exit 1; }

The patience in that loop is earned from being burned: a freshly-restarted service doesn't answer instantly, so a naive single check right after the restart fails a perfectly healthy deploy. The retry-with-backoff waits for the service to actually bind its port and answer a real request, and only calls the deploy failed if it stays silent past a real deadline. Non-critical steps that follow — purging the edge cache, regenerating a sitemap — are allowed to hiccup without failing the whole deploy, because they're recoverable and the health of the app is not.

The through-line across all of it is that the deploy verifies reality rather than assuming success. It doesn't trust that installing the files worked; it validates. It doesn't trust that the reload was fine; it curls the live endpoints and waits for a real answer. Green means the actual system answered a real request, not that a script reached its last line without erroring — which is the standard I try to hold everything to that touches production. A status of 'done' is a claim; a 200 from the live endpoint is a fact; and the deploy is built to keep checking the claim against the fact so I don't have to stand over it hoping.