At least once is not exactly once

I had a task that sent a new customer their welcome email, and it was set to acknowledge late. It tells the broker it is done only after the work finishes. If the worker goes away first, the broker still holds the task and hands it to another worker. For a welcome email that sounds right. It is worth retrying.

Inside the task the order was send, then acknowledge. Between those two steps there is a gap, and the task recorded nothing in it. A worker killed in that gap has already sent the email but has not told the broker. The broker does what it promised and delivers again. Another worker sends the email again, and the customer gets two.

At least once is the promise. Exactly once was never on offer, and my task read as safe only because I had not looked at the gap.

I found this while cleaning up after a different problem. My Celery workers ran as fourteen systemd units, and none of them set a stop behavior. So systemd used its default: kill the whole control group, with a 90 second timeout. SIGTERM went to every pool child at once, and each running task died with WorkerLostError and signal 15. A failure handler had been filing that under deploy noise.

When I counted from the worker journals, there were 125 deploy kills in seven days, and my run log held zero rows for them. I made the workers drain on deploy, and that change merged first. Then I wrote a four-step follow-up plan for the damage those kills could do, because a kill is only noise if the task it hits is safe to run again. Step two was the welcome email.

The plan named two email tasks that acknowledged late and kept no record of having sent: the welcome email and an internal alert for new leads. git blame showed no reason why either one had acknowledge-late turned on. It had simply been there, and I had been reading it as the careful setting.

One option was to turn acknowledge-late on everywhere. I rejected that for long tasks. On Redis, an unacknowledged task is redelivered after the visibility timeout, an hour by default. Any acknowledge-late task that ran longer than that would run twice while perfectly healthy, and a redelivered paid task would spend its API budget a second time.

When I read the code, the problem was smaller than the plan had said. Both email services catch every send error and return False. So a retry from inside the task can only fire before a send, never after one. The only path to a duplicate was a kill in the gap between the send and the acknowledgment.

Then I left the lead alert alone on purpose. A duplicate internal alert is harmless. A lost one is a missed lead. For that task, acknowledge-late with no marker is the right trade. The welcome email was different, because the customer is the one who sees the duplicate.

The fix was a sent marker kept in Django's cache with a thirty day expiry. The task checks for it first, before it even looks up the submission in the database, and returns early on a hit with a log line saying it already sent and is skipping the redelivery. It writes the marker only after the send returns True. A run that failed, or did not send, leaves no marker and stays retryable, which is right, because that customer never got their email. I kept acknowledge-late.

# the order inside the task (names simplified)
if cache.get(marker_key):
    log.info("already sent; skipping redelivery")
    return False
submission = load_submission(submission_id)
sent = send_the_email(submission)
if sent:
    cache.set(marker_key, True, timeout=30 * 24 * 3600)  # thirty days

Three new tests cover it: two runs send once, a run that did not send leaves no marker, and different submissions stay independent. The eight existing welcome-email tests still pass. When I removed the marker write, the duplicate test went red. That is all the proof I have. It has not been exercised in production, because that would take a real kill in the middle of a send.

I wrote the expected benefit down as small, and the change describes it in one line.

This closes a rare but customer-visible failure.