A dry run that spent money

I was getting ready to switch a content gate from report mode to enforce mode. In enforce mode the gate stops pages from publishing, so it is the kind of change I want to rehearse first. The publish command has a --dry-run flag for exactly that.

I read the command before I used it. In its handle method, the harm gate runs first, and the if dry_run: return comes after it. So a dry run did everything the gate does, then stopped just short of publishing. The gate was the expensive part.

So I did not use the dry run to preview the flip. I replayed the gate in a separate shell with its two model calls stubbed out, and I wrote the reason into the readiness report. I did not use --dry-run, because it still makes paid LLM calls.

The same readiness check turned up a second gap at the same time, empty and placeholder pages passing the quality score, which is its own story. That gap let bad pages through. This one turned the tool for checking safely into a way to spend money and write to production.

For each page on a site marked as naming real entities, the gate asked two questions. Is the model available? Is this page grounded? Both went through the same prompt runner, which makes a paid call to a Sonnet-class model and saves a prompt-run row to the shared production database. A dry run of a site full of real places paid for every page on it.

Then I went looking for every write a dry run could reach. The quality score save was already guarded. So was the log of harm verdicts. The prompt runner calls were the only paid calls and the only unguarded writes: one leak, in one place, behind a flag whose name promised there were none.

I do not know what it cost. No one measured how many rows a dry run created, and I have no dollar figure for the leak. I found it by reading the code, not from an alert or an invoice.

The runbook for the flip got a warning in step 5, written that same afternoon:

Do NOT use publish_programmatic_pages --dry-run to test this. Today it still makes paid LLM calls and writes PromptRun rows.

Its verification step avoided the command entirely and read the resolved gate mode with a one-line shell call instead. The flip itself was a single flag row with a rollback step.

A warning in a runbook protects only the person reading that runbook. Anyone who reaches for --dry-run on another day, for another reason, never sees it. So the warning was a patch meant to last until the code changed, and the code changed the same day.

The gate takes a new setting, allow_llm, which is true by default. The publish path passes allow_llm=not dry_run. When it is false, the gate skips both model calls and counts the page as needing a model check. The free hard block on unsourced quotes still runs, because it costs nothing.

A dry run now prints how many pages would need a model check and says plainly that they are not verified:

[DRY RUN] would need LLM check: N page(s) ... NOT verified

Those pages are never reported as passes, and the help text for --dry-run now names the rule.

Keeping the default true was deliberate. Every caller that is not a dry run behaves as before. The cost is that a dry run shows real-entity pages as eligible, with that warning, even though enforce mode could still block them on the real run. I would rather a rehearsal admit it did not check something than imply that it did.

Six tests cover it, at the gate level and through the full command. A dry run makes 0 prompt calls, writes 0 prompt-run rows and 0 action-log rows, saves no quality score, and opens no remote connection. A real run still calls both checks.

Then I put the old bug back on purpose by forcing allow_llm to true. Two dry-run tests turned red, and the failure read:

a dry run made a paid LLM call
assert 3 == 0

Forcing it to false turned 3 real-run tests red. The tests patch the prompt runner, so the tests never called a model either.

Two things are still open. After the deploy, a production dry run should add 0 prompt-run rows, and a follow-up is filed to confirm it. And a separate orchestrator also calls the availability probe. I have not looked at what its dry run does.