An alarm on the wrong thing drowns the signal

My new alarm flagged ten call sites on production as mispriced, and none of them was a real gap. I had predicted it, in writing.

One change earlier I had built a cost baseline: a read of every model call, grouped by the place in the code that makes it, priced through one pricing module. It introduced a status called mispriced for any model id that module priced at zero dollars. A zero usually means the pricing table does not know the model, and an unknown model silently costing nothing is exactly the blind spot I wanted closed.

That same change had a not checked section, and in it I deferred a split between genuinely free local models and paid model ids with no price. The note read:

Refining MISPRICED to split genuinely-free local Ollama models from unpriced-paid Anthropic ids - deferred (Phase C/E taxonomy); for now both are conservatively flagged.

Conservative was the right word. It was also the word that put ten false alarms at the top of the board.

The flagged sites ran two local models through Ollama on my own hardware, gemma3:27b and qwen3:32b. A local model has no API bill. Zero dollars is the correct price for it. The alarm could not tell a correct zero from a missing one, because to the rule they were the same number.

The damage was to the ranking. The baseline exists to show me where model spend goes, ordered from the largest call site down. Ten false flags were enough to bury that table. It led with work that cost nothing, and the paid call sites I needed to look at sat underneath a pile of correct zeros.

The rule needed one more fact, and the data did not carry it. The ledger reader grouped calls by code location and model, and nothing else. Given only a model id, the cost layer had no way to know whether an unpriced model came from the paid provider or from a local runtime.

So the provider had to travel with the row. I added it to the grouping in the reader and into each per model detail, with unknown as the fallback when it is empty. That change was three lines and purely additive. The older consumers of the same ledger, a spend monitor and the gross profit calculation, were left alone.

Then the definition got narrow. A call site is mispriced only if the pricing module cannot resolve the model family and the call's provider is the paid one, spelled either claude or anthropic. In the code the paid providers are one set:

_ANTHROPIC_PROVIDERS = {'claude', 'anthropic'}

Anything else that cannot be priced goes into the list of local models, still adds zero dollars, and is labeled local on screen. Real gaps now print as unpriced with the paid provider named, so the two cases look different when I read them. A call site that is entirely local still counts as measured, with its real call counts. It does not alarm.

The same read only run against production went from ten mispriced to zero, and the top of the table was paid work again.

The alarm is a small piece of a larger cleanup. My production code held 153 hardcoded model id strings, each one a place where a model choice was frozen into a literal. I added a test that walks the app code, skipping tests and migrations, counts the lines that match a model id pattern, and fails if the count ever rises above 153. The baseline is whatever the test's own walk computes, so anyone can reproduce it.

The pattern is narrow on purpose. A plain grep for the claude prefix would also match tool and package names that share it, and a count full of those would make the ratchet meaningless. The pattern requires a real model family word followed by a digit. A second test fails if the count drops more than 25 below the baseline. That forces whoever lowers the count to lower the baseline in the same change, so the guard never drifts into a state where it cannot fail.

I seeded a literal to check that it bites: 156 against 153, red. On the clean tree, green. The rule is only that the count never gets worse. There is no target number. The next round of work built a central place to choose models and took the count from 153 to 139, and the baseline only moves down.

One more guard checks that every model id I plan to route to prices above zero. Its negative control is a made up model id, which must price at exactly zero. That keeps the original risk written into the tests. An unknown id costs nothing and says nothing.

That guard has a limit I want on the record. A nonzero price proves the pricing table knows the model family. It does not prove the rate is current. A newer model priced at its older family's rate would still pass. Refreshing the rates is later work, and until it lands this check can only tell me a model is known.

An alarm has to know the exact class of thing it watches, and here that class was defined by a fact the data did not carry yet. Writing the overreach down when I built it made it fast to find. It did not stop ten correct zeros from hiding the numbers the alarm was built to show me.