Dead is not asleep
A liveness alarm kept telling me to revive a laptop that was healthy and asleep, until I changed what the alarm counts as dead.
By Andrew Pyle
A closed laptop lid and a crashed process send the same signal, which is no signal at all.
01What silence means
The tripwire knew one reason for a quiet machine
When a laptop stopped talking to my monitor, the monitor concluded the worst, every time. I keep a tripwire on the launchers that start work on my Macs. Each launcher sends a heartbeat, and if the dispatch launcher goes quiet for 20 minutes, the tripwire calls it dead.
Every link in the chain after that worked as designed. The tripwire raised a launcher-dead alert. A remediator picked the alert up and filed an approval card asking me to revive the MacBook Air. The card waited in my queue for a decision.
The remediator could not fix this one itself. The registry marks this class of failure propose-only, because production cannot restart a launchd agent on one of my own Macs. The only remedy it can offer is to ask me. Another class in the same registry restarts a dedicated canary worker on its own and confirms the restart from the logs. The launcher was never going to get that treatment, so every alarm cost a human decision instead of a log line.
And the laptop was usually just asleep. Closing the lid stops the heartbeat. To the tripwire, a closed lid and a crashed launcher looked the same.
It also did not give up. While a daemon stays stale, the tripwire rechecks about every five minutes. A four-hour dedup window keeps it from posting a fresh alert on every check, but the count on the row keeps climbing, and after four hours it posts again. A machine I have retired keeps alerting until I delete its heartbeat row, which is how I tell the system a host is gone. A card that keeps arriving for a healthy machine teaches whoever reads it to dismiss it.
02Asleep is a state
I taught the alarm when silence blocks work
Thresholds were already set per daemon. The launcher used 20 minutes and the crawl host used 90, because their cadences differ by four and a half times. One global number had been wrong before. This change made the alarm depend on the machine as well.
There is now a short list of sleep-tolerant Macs, and by default it holds only the MacBook Air. For a Mac on that list, a stale dispatch launcher counts only when a queued ticket targets that Mac, which is the case where work genuinely cannot start. With nothing queued, the Mac goes onto a new asleep list in the result and the loop moves on. The rule covers only the dispatch launcher. The other services on that laptop keep their normal staleness checks.
The version check waits too
The same skip covers the version check. A laptop with its lid shut is running no code, so it should not be judged on which code it runs. Without the skip, a deploy while the Air slept would have produced a page saying it was running stale code. Its version now waits until it wakes and reports again.
The Studio keeps the plain alarm
The Mac Studio did not change. It keeps the plain 20-minute alarm, and the comment above its test says why:
The always-on Mac Studio: silence alone is a finding.
What the tests pin down
One test makes the laptop's launcher stale by pushing its last poll back ten hours with a direct update, so the model's automatic timestamp cannot reset it. With nothing queued, it asserts no stale count, no stale version, the Air on the asleep list, and no alert sent. A second test adds a queued ticket aimed at the Air and asserts the alarm fires. An older test that used a generic host now uses the Studio, because that is where the old meaning still holds. The suite I ran locally passed 65 tests. The change touched two files, 70 lines added and 2 removed, and I merged it about 51 minutes after opening it, a little after 11 PM.
03A new blind spot
The quiet has a cost of its own
The new rule infers sleep from silence. Nothing in the code reads the laptop's power state, so asleep means silent with no queued work. That is also what a dead launcher with no queued work looks like. On the Air, that launcher can now die quietly, and nothing will say so until work is aimed at it or I notice. I did not add a second, longer silence alarm as a backstop.
When the alarm does fire for a queued ticket, it tells me work cannot start. It does not wake the laptop. Someone still has to open the lid.
The habit I am keeping: before an alarm asks a person to act, it should know what the silence it reports actually blocks. For a machine that is allowed to sleep, quiet is normal, and the event worth a card is waiting work that cannot start. I traded a false alarm that arrived constantly for a blind spot that may rarely matter, and that trade is written down here so it stays a decision and never turns into a surprise.
Related
writing
A close that would not stick
I was about to clean up a third of my task board, and the code that filed those rows would have put them all back.
writing
A queue with no backoff floods on every tick
My approval queue filled with the same pull request on every tick, and the thing it kept trying to merge could never merge.
writing
A dry run that spent money
The flag meant to make a rehearsal safe was making paid model calls and writing real rows.
writing
Active was measuring the wrong thing
My overnight builder thought I was awake from 1 to 7 AM, because the signal it trusted for awake was being touched by something that was not me.
writing
The publishing pipeline that lied to me for four days
A green log, a stale site, and the difference between a job that ran and a job that did the thing.