A test with a clock in it
A test passed cleanly in review, then failed on every build three days later, and nothing in the code had changed.
By Andrew Pyle
The only thing that changed between the green build and the red one was the date.
01Monday, 7 AM
Main went red without a commit
On the morning of Monday, October 5, an unrelated pull request failed its build twice. The change in it had nothing to do with the test that broke. The last green build on main had run shortly before. So my first suspicion was the ordinary one: something in that branch, or something that had landed just ahead of it, had broken a module it never touched.
The failing test was the happy-path test for a command that writes a context pack. The error was "Database access not allowed." That message points at the database, but the database was only the place where the failure came to the surface.
The command is small and strict. It builds a static pack of context for coding tools that cannot query my data directly. It pulls from public-safe endpoints through a proxy, stamps each section with its freshness, scans the result for secrets and personal data, and fits it into a budget of 6,000 tokens. The first live run used 368.
02A dead-man that worked
The command was right to refuse
The command has a dead-man built in. If a required section is empty or older than a maximum age, 72 hours by default, it blocks, raises a high-severity alert, exits with an error, and writes nothing. I built it that way on purpose, because a false all-clear is the failure I trust least. The docstring puts the rule in one line:
A stale/empty pack must never read as success.
The alert is what touched the database. In the test, the command decided its data was stale, took the alert path, and tried to record the alert. The test does not allow database access, so the failure that printed was a database error. The staleness that caused it sat one layer down and never appeared in the output.
So the production code was doing its job. The test was what had changed, and it had changed without anyone editing it.
03Two clocks
The fixture froze time and the command did not
The test stamped its fake payloads at a fixed moment: noon UTC on Friday, October 2. The feature merged at 4:42 PM Central that same day, nearly ten hours after that stamp. CI ran green, because the payloads were well inside the 72-hour window.
The pure function that assembles the pack takes the current time as an optional argument. Its own unit tests pass a fixed time, so they stay deterministic. The management command does not pass it. It calls the function without a time, and the function falls back to the real clock. I never chose which clock the command would read; the default chose it.
That left one side of the comparison frozen and the other side moving. The test stayed green through review, through merge, and through about 62 hours of builds. At noon UTC on Monday, 7 AM Central, the frozen stamp turned 72 hours old against the real clock. The command saw stale data and refused, and every build after that failed. It went off on the first workday morning after a weekend, which is about the least convenient time for a build to start failing.
04One clock per comparison
The test now stamps its data at real now
I changed only the test. A new helper returns a fake fetch whose payloads are stamped at whatever time it is given, and the two command tests now give it the real current time. The pure function tests keep their fixed time, because they pass it in explicitly and both sides of their comparison agree. I folded the old fresh-data helper into a shared payload builder so both clocks produce the same shape. The whole change was 16 lines added and 3 removed, in one file.
I left a comment in the test that names the trap, including the exact moment it went off:
# The command judges staleness against the REAL clock (no `now` injection),
# so stamp the fake payloads at real now. Stamping them at the fixed NOW
# turned this into a time bomb: it went red once real time passed
# NOW + DEFAULT_MAX_AGE_HOURS (2026-10-05 12:00 UTC).The test file passed 10 of 10 after the change and failed before it. Main had been red for about two hours.
A second test, passing for the wrong reason
While I was in there I found a second test passing for the wrong reason. It checks that the command blocks when a secret reaches the pack. With the frozen stamp, it had been blocking on staleness as well as on the planted secret. A check that confirms the wrong thing is how my publishing pipeline once lied to me for four days. After the change it blocks on the secret alone, which is the thing it claims to test.
What I did not do
Two things I did not do. I did not give the command itself a way to accept the current time, so it still reads the wall clock and its tests still have to stamp data to match. And I did not look for other tests that pair a fixed timestamp with a real-clock threshold. If one exists, it has a date on it too, and I do not know which date.
Related
writing
A limit that was never enforced
Eighty of my scheduled tasks ran on a time limit different from the one they declared, because the loader quietly threw it away, and the first plan I built on those numbers was wrong.
writing
A deploy should drain, not kill
A routine deploy quietly ate a day of work, and the monitor that was supposed to notice reported nothing wrong.
writing
A status the page never learned
The tasks page went white the day a triage used a status value the client had never been told about.
writing
A cold cache is not free
My feature-flags page felt instant once I cached it, and the first request to each fresh worker after a deploy still paid about fifteen seconds.
writing
A 200 is not the same as access
My leak probe failed with seventeen leaks on its first real run. Every one of them was the same page that was never a leak.