I assumed the bigger model was the better judge

I had never asked what it should cost to have a model decide whether a page is good enough to index, because I had never chosen the model. My content judge reads a page and decides whether it earns its place on the site. It runs inside my content scorer, inside the gate I use before publishing, and as a command I can run by hand. In all three places it had no model of its own. It rode the platform default, which was Sonnet 4.5.

I found out what that cost when I built a cost baseline by call site. My earlier tracking could only see spend by model and lane, so it could tell me which model was expensive but never which line of code was doing the spending. The new baseline ranked every call site through one pricing table. The judge came out at about 88 percent of the bill.

The bill itself was small. It was about $6.38 over thirty days, and 88 percent of a small number is still small. The baseline also declined to call the judge measured. Its roughly 470 calls had all landed on a single day, below the floor it needed to trust a rate, so it labeled the line unmeasured instead of guessing. I would rather have that label than a confident wrong number.

The comparison I needed already existed. Earlier I had run a bake-off: the same judging prompt sent to Sonnet 4.5, Haiku 4.5, a free local classifier, and a small encoder model, all scored against my own labels. The whole thing cost about $1.75.

I labeled the pages blind in a plain labeling page. The first round was 100 pages, 59 I called thin and 41 I called substantive, split into 50 for scoring and 50 for calibration. The second round was 100 fresh pages, 82 thin and 18 substantive. Along the way I changed the question. A four-way verdict became a yes or no: does this page earn indexing on its content alone?

On the first round's scoring slice, the four judges came out like this:

substance task, round 1 scoring slice (n=50)

judge              macro F1   $ per 1k calls   median latency
Sonnet 4.5         0.557      3.24             1,139 ms
Haiku 4.5          0.700      1.43               733 ms
local classifier   0.818
small encoder      0.578

Sonnet's macro F1 was 0.557 and Haiku's was 0.700. The free local classifier scored 0.818, the best of the four. Haiku was also cheaper, $1.43 per thousand calls against Sonnet's $3.24, and faster, with a median of 733 milliseconds against 1,139.

The change itself was about 22 lines. A new setting names the judge's model, defaulting to Haiku 4.5, and the three call sites read it. The hand-run command can still take a different model per run. The judge and my content generators sit on separate settings on purpose, so moving one never moves the other. Going back to Sonnet means changing a single secret value, with no deploy. A test pins the default and checks that the call site follows the setting, and it lives in the folder CI actually runs.

I did not run a fresh bake-off first. Two blind rounds against my own labels seemed like stronger evidence than a new sample. The saving is about $3.70 a month. The reason to move was the direction of the errors. The check I wrote down is the next spend window: cost per judge call down about three times, with the Haiku model name on the rows. Later the bake-off's reference arm was renamed from sonnet reference to prod reference, because production no longer runs Sonnet for this.

So the reflex was wrong twice. The default was more expensive, and it was worse at the part of the job I care about most. I picked it by never picking, and bigger felt careful until the labels said otherwise. The question I have not closed is the one the review left open: whether the free local classifier, which beat both paid models on the first round, should be the one judging.