The smoke test that only checked anonymous

The smoke test in the deploy pipeline for one of my apps waits ten seconds and then runs two commands. The first fetches the home page. The second fetches a health route on the API. Each passes if the server answers without an error. If the production address is not set, the step skips politely and exits clean.

# post-deploy smoke test, after a ten second wait
curl -sf -o /dev/null "$PRODUCTION_URL/"
curl -sf -o /dev/null "$PRODUCTION_URL/api/health/"

Neither request carries a session, and neither route sets a login cookie. So the only thing the check asks the server is whether it will answer a stranger. On Saturday, September 26, it would, while every logged-in user got a Cloudflare 502 on the server-rendered pages, the dashboard and onboarding among them.

The app itself was healthy. Asked directly on the server, with no proxy in between, it answered normally on its local port. The failure lived in the layer in front of it.

There was a second trap waiting for anyone who reached for curl to diagnose it. An anonymous request for the dashboard returns a 307, the redirect to sign in, whether the bug is present or not. That 307 looks like a working app, the same way a soft 404 looks like a real page. The only check that could tell the difference was loading the dashboard in a browser while logged in, so that is what the last step of my runbook says to do.

nginx logged the real cause in one line:

upstream sent too big header while reading response header from upstream

The location that proxied to the Next.js app had proxy_buffering off and no proxy_buffer_size, so the default header buffer of 4k or 8k applied.

A logged-in response carries a Set-Cookie header holding the Supabase session, and that cookie had grown. Enabling Google single sign-on adds Google's provider tokens to the session. Once those were in it, the header crossed the ceiling and nginx refused the upstream response. Anonymous responses carry no such cookie, so they stayed under it. That is why the outage sorted so cleanly by login state.

The change itself is small: three lines inside the location and one at the server level.

# inside location /, after proxy_buffering off;
proxy_buffer_size 32k;
proxy_buffers 8 32k;
proxy_busy_buffers_size 32k;

# server block only
large_client_header_buffers 8 32k;

The runbook for it was six steps: back up the config, add the lines, run nginx -t as a hard gate, reload only if that passed, then confirm a logged-in dashboard renders. I applied it on the live server by hand.

Writing it down properly caught a mistake in my own fix. The first version put large_client_header_buffers next to the proxy lines. That directive is valid only at the http or server level, and inside a location it fails nginx -t. The shared snippet I committed holds only the three proxy lines and documents the fourth for the server block.

I codified the fix in two places. The same morning, a shared snippet and a written standard went into my central deployment docs: any nginx site in front of Next.js with Supabase server-side auth must enlarge its proxy header buffers. Next to the standard went an audit one-liner that lists site configs proxying to a local Node port with no proxy_buffer_size set. Those configs are managed by hand on each server, which is why the audit has to run there. Any file it lists will 502 for logged-in users once its auth cookie grows, and enabling an OAuth provider is enough to make that happen.

The app's own repo caught up sixteen days later, when its nginx template gained the same values with a comment pointing back at the incident. A rebuild from the repo now keeps logged-in pages working. A routine deploy still reloads whatever config is already on the server, so the repo copy and the live copy meet only on a rebuild. Like the gap between deployed and served, the repo holding a fix does not tell me what the server runs. That gap is narrower than it was, and it is still a gap.

The part I have not changed is the one in the title. The smoke test still makes two anonymous requests. The buffers, the snippet, the standard and the template are all fixed, and the check that called the site fine on the day logged-in users could not use it would say the same thing again. A smoke test that never signs in proves the server will answer a stranger. It says nothing about whether a logged-in person can get through, and on this app that was the only question that mattered. I hold my features to the rule that built means exercised, and my own smoke test never exercised the logged-in path. The missing step is one that signs in and loads a page only a logged-in visitor can see.