Stop indexing what you copied
My sitemaps looked healthy because they counted pages I had not really written.
By Andrew Pyle
Over one week I cut three sites down to the pages they could defend, then made the builds refuse to drift back.
01A question about main
What would a rebuild ship?
I wanted to know what a rebuild of my historic places site from main would ship. The answer was worse than the live site. The live site indexed 2,147 place pages, with a sitemap of 2,210 URLs, and every other place page was live but marked noindex,follow. That rule, though, lived only on a branch that had never been merged. Main still used an old floor: a place needed a name, coordinates, and a state. A local build of main produced 90,072 sitemap URLs. A rebuild would have indexed about 90,000 thin pages.
Nothing in the deploy would have caught it. The host does not build the site from git. It is a static-assets deploy that I push by hand, and the host records no git reference, so even which commit was live was an inference from timing: the last deploy landed four minutes after the branch tip commit. The index depended on me remembering which branch I had built.
It had gone wrong once already. In September a commit widened the indexable place set from 2,148 to 36,943, and I reverted it the next day.
02One rule, one guard
Putting the rule where the build can see it
First I moved the rule onto main and into one function, placeIndexable. The sitemap, each page's robots meta tag, and the state hub counts all call it, so they cannot disagree, which is what an honest sitemap needs. Then I added a postbuild guard and the repo's first CI. The guard checks that the set of place URLs in the sitemap equals the set of built place pages without noindex, that the count is above zero, and that it is under a hard cap, 2,500 at first.
Its output is three numbers and the cap. I proved the guard works by breaking it. Reverting placeIndexable to the old floor made the build fail and exit 1. Adding one noindexed page to a copy of the sitemap made it fail on that one page.
built place pages: 90176; sitemap place URLs: 2147; place pages without noindex: 2147; cap: 2500
# placeIndexable reverted to the old floor
FAIL: the sitemap lists 90009 place URLs, above the hard cap of 2500.
# one noindexed page added to a copy of the sitemap
FAIL: 1 sitemap place URLs render noindex or are missing03Four of four
The pages that survived were copied
That fixed the count. It did not fix what the count was counting. I sampled four of the 2,147 indexed place pages. In all four, the About this place text matched Wikipedia word for word, and on those pages that text was the only narrative there was. Google already holds the original of copied text, so a copy is close to the weakest page a site can offer. I had already seen what a cluster of weak pages does to a whole domain.
So the quality gate got stricter. A place now indexes only if it is valid, has an original summary marked verified, that summary is at least 250 characters, and its source is not Wikipedia. The Wikipedia summary no longer counts for anything. Pages with a Wikipedia source show only a link, Read about this place on Wikipedia, with a CC BY-SA 4.0 note. Of 2,335 pages with a Wikipedia summary, 0 render it now.
That left 28 place pages, none of them from the old indexed set. Each fact in their summaries was checked against a verbatim span of that place's National Register of Historic Places nomination form, mostly National Archives PDFs. One source is a National Park Service PDF, so its label says National Park Service, which keeps the label true. The guard's cap dropped from 2,500 to 100, and its line now reads sitemap place URLs: 28. No page was deleted. All 90,176 built place pages still exist.
This had a real price. Up to 340 of the site's 342 clicks over the previous 28 days had landed on place URLs, and all 2,147 of those pages went noindex,follow. I accepted that trade on purpose. The new sitemap holds 91 URLs: the 28 verified places and 63 hubs.
04The same move twice
Two other sites, two other floors
I made the same move on two other sites, each with a different floor.
A floor of three on filmed_at
On filmed_at the thin pages were templated rather than copied. A film page indexed if it had at least two distinct filming places. Pages with only two places read close to a template, and in seven weeks of search data the indexed film pages had 0 clicks. I raised the floor to three. Sitemap film URLs fell from 4,188 to 1,853, and noindexed film pages rose from 12,515 to 14,850. The two-place pages that went noindex had 228 impressions and 0 clicks in 28 days. The methodology page prints the constant, so it now reads at least 3 distinct filming locations.
Its guard, assert-index-floor.mjs, works out the floor from the film data on its own instead of importing the constant, so a wrong constant cannot vouch for itself. Run against the old build output, it exited 1 with 4,670 problems. With the robots tag removed, it found 14,850.
This blog, where the error ran the other way
This site, my personal one, got the same treatment, and here the error ran the other way. A prune review re-indexed one post on building REST APIs with Django REST Framework, a real practitioner post with real code that an earlier sweep had wrongly caught. The same review set 14 thin stubs under 100 words to noindex. Here noindex is a database field that the deploy turns into an nginx X-Robots-Tag header, and the sitemap is rebuilt from the same data.
05Still open
A guard only covers the path it runs on
The prune happened once. The guards keep it in place. A build that counts its indexable pages and refuses when the number is wrong stops a copied page from leaking back in on some later day when I am not looking.
A guard only runs where the build runs, though. The filmed_at repo had no CI on main, and I have not confirmed that its live deploy uses the build command that runs the guard. If it does not, the check never fires there. The historic site deploys by hand, so its 91-URL sitemap is live only once I push it, and I have not recorded that live check passing. A second, larger wave of places with longer summaries is planned and not built. Each of those is a place where the rule exists and the enforcement is still my memory.
Related
writing
A zero page crawl is not a baseline
I shipped a batch of SEO fixes, and a clean reconciliation pass told me none of them had landed.
writing
A neutral default ships garbage
My page scorer gave an empty page a passing grade, because every missing signal quietly scored as average.
writing
Re-dating a drip without hiding what's already live
A date-based publishing drip is elegant until you re-date a page that's already indexed. Then it quietly un-publishes it.
writing
A billion indexed pages is a direction, not a page count
I keep an absurdly large number as a north star — but the number isn't a target to hit by publishing a billion things. It's a forcing function that makes you solve the problems that only show up at scale, and it fails the moment you treat page count as the point.
writing
A sitemap should only list pages you actually want found
A sitemap is a set of recommendations you make to a search engine. Listing pages you've told it not to index, or that redirect elsewhere, is contradicting yourself — and a sitemap that contradicts itself teaches the crawler to trust you less.