My blog was invisible to Google (and it was my fault)

The check is one command. Ask for a page the way Googlebot does, and read the title it gets back:

$ curl -A "Googlebot" https://andrewjpyle.com/writing/agentic-development \
    | grep -o "<title>[^<]*</title>"

# what I expected:
#   <title>What I mean when I say agentic development — Andrew Pyle</title>
# what I got:
<title>Writing — Andrew Pyle</title>

That is not the essay. That is the title of the /writing index — the list page. Every one of my sixteen essays, asked for by a crawler, returned the index shell: wrong title, wrong canonical, and — worse than either — no body at all. In a real browser the same URL rendered flawlessly. The bug was invisible exactly where I was looking and present exactly where I wasn't.

A page that renders in your browser and a page that exists to Google are two different claims. A single-page app quietly lets the first be true while the second is false.

My essay page fetches its own body. The component mounts, and inside a React useEffect it calls the API for the post:

// WritingPost.tsx — the body arrives asynchronously
useEffect(() => {
  getPost(slug).then((p) => setPost(p));   // runs in the BROWSER, after mount
}, [slug]);

Here is the trap. To give crawlers real HTML, I prerender each route at build time with React's renderToString — it walks the component tree and returns a string of HTML. But renderToString renders once, synchronously. It does not mount, it does not schedule effects, and it never runs a useEffect. So at prerender time the fetch above simply never fires. The component renders its initial state — no post yet — which is the loading spinner. That spinner is what got written to disk and shipped to Google.

The confusing part was that my /work project pages — same site, same prerender script — were perfectly crawlable. The difference is where the content lives. A case study's body is static data inside the component: it's right there in the module, so renderToString renders it immediately, no async step. An essay's body lives in the database behind an API call. Same rendering engine, opposite outcome — entirely because one waits on a fetch and the other doesn't.

That's the real lesson under the bug: server-side rendering only captures what's synchronously available. Anything you fetch after mount is invisible to the prerender, and therefore invisible to any crawler that doesn't execute your JavaScript.

The fix is to make the essay body synchronously available at build time — without giving up the live fetch for real readers. Three parts:

// initial state now comes from the build-time snapshot, not null
const seed = writingSeed(slug);
const [post, setPost] = useState(seed ?? null);
const [loading, setLoading] = useState(!seed);

The reader experience doesn't regress. On mount, the page still re-fetches the live version in the background, so anything I've edited since the last deploy shows up fresh. The snapshot only has to be good enough for the crawler and the first paint; the network fills in the rest. Nobody blanks to a spinner over content that's already on screen.

Adding a fetch to the build introduces a new way for the build to fail — a live API any database hiccup can take hostage, blocking an unrelated deploy. So fetch-writing.mjs degrades instead of dying. It retries three times with backoff, treats any HTTP 5xx as retryable, and aborts each attempt after twenty seconds rather than hanging the build.

When it still can't get a clean answer — a timeout, an unexpected shape, zero successful fetches — it calls one keep() function that logs how many essays are already committed and exits zero. The build carries on with the last-good snapshot that's checked into the repo. A stale essay body for one deploy is survivable; a red build blocking every unrelated change is not.

There's a subtler guard too. If the API answers with a valid but empty list — 200 OK, zero essays — the script refuses to overwrite good committed data with nothing. A silent empty payload is exactly the "success" that would wipe every essay off the site while the status lights stayed green.

Baking the essays in isn't a special case I invented for this bug. It's the fifth of five fetch steps the build runs before Vite ever compiles: dev activity, portfolio, network, shelf, then writing. Each freezes live API data into a committed JSON file so the prerender has real content to render synchronously. Seen that way, an essay is just another external data source, and the fix stopped feeling like a hack.

The hero stats on my homepage work the same way. Those numbers are fetched from a separate service at build time and frozen into the bundle, not fetched live, so a crawler and the first paint both see a real figure instead of a placeholder. The rule underneath: anything a crawler needs to see has to exist at build time, because build time is the last moment my code runs before the HTML is frozen and shipped.

This bug is one instance of a more general trap I keep relearning: you have to know exactly which layer a crawler actually sees. Two ways that bit me on this site.

First: for a long time my instinct was to fix SEO in Django — the view, the template, the contrib sitemap. But a React SPA is the real serving layer here. Every fix I made in a Django view was inert in production, because that code path isn't what a crawler receives. Days spent editing a file Google never reads.

Second: even after the fix, a shared prerender/cache layer can serve a stale render for up to 24 hours while the CDN cheerfully reports the response as fresh. The staleness lived a layer deeper than the cache header I was trusting. The only reliable check is to fetch the bare canonical URL as a bot and read what actually comes back — not a cache-busted variant, which is a different key and always looks fresh.

After the fix, the build prerenders every essay with its real body — verified page by page, each one now carrying its own title and canonical instead of the index shell it used to hand over. The command that used to return <title>Writing — Andrew Pyle</title> now returns the essay's actual title, with the actual prose underneath it.

The uncomfortable footnote: those essays existed and were good for months, and Google was reading a blank the entire time. No amount of writing would have moved anything, because the writing was never the page under evaluation. It's a reminder I'd put above any SEO checklist — verify the crawler's-eye view, not the browser's. They are not the same page, and only one of them ranks.

None of this is exotic. Prerendering a single-page app is a solved problem in principle. But the failure mode is silent by construction: it looks perfect in the one place you naturally check, and broken only in the place you have to go out of your way to see. The fix wasn't clever — bake the data in so it's there before the async step. The discipline was remembering to look at the page the way a machine does, not the way I do.