$4 ISR writes, GPTBot, and why software engineers are still valuable

There is a lot more to software engineering than writing code. In this post I want to go over a situation that happened with IndieDex.gg, and how I still have to make a lot of judgements and guide the LLM towards the right answers.

It’s an update on IndieDex and what I’m doing with it to improve SEO, AEO and GEO. It’s also about why software engineers are still valuable: even with AI getting better at coding, it doesn’t get better at understanding the needs of the business.

Why ISR writes took off on a 960K-page site

Numbers are from IndieDex’s own Vercel usage, firewall traffic, production logs and Search Console data, October 2026.

IndieDex is a game discovery site. Most of its pages are SEO pages generated from the catalog: a detail page and a “games like X” page for each of about 120,000 games, in four languages. That is roughly 960,000 pages. We can’t predict which of them will rank. Google surfaces games that look random from the outside, so every page has to be fast and crawlable.

In early October, one line on the Vercel bill took off: ISR writes. Those are the writes that store a statically rendered page in Vercel’s cache.

Vercel bills ISR writes in 8 KB “write units”, at $4.00 per million in its US East region. A write only happens when new or changed content is stored: revalidating a page that didn’t change costs nothing, according to Vercel’s ISR pricing docs.

A few dollars a day isn’t a crisis. It is a signal, though. It meant the cost was now tied to something that grows: the number of distinct pages that get visited. On a 960K-page site, that is what turns a side project into a bill.

This is a follow-up to IndieDex Log 1, where AI crawler storms pushed daily costs as high as $15 and ISR with a months-long cache was the bet.

How I worked through it with Claude Code

I worked through this with an AI coding agent (Claude Code), which also spun up research and verification sub-agents along the way. What I want to show is the shape of the conversation: what I asked, what the AI got right, what it got wrong, and where my questions changed the outcome.

1. Anchoring the timeline to the Oct 7 merges

My first question was simple:

“so since october 7th the ISR writes have gotten a lot more expensive, what caused that?”

The AI’s first answer was plausible and partly right. A reviews query used a short cache profile, which had pinned the cache lifetime of the game pages that are built ahead of time to 60 seconds, and a fix for that had already shipped. I pushed it back to the date:

“take a look at the merges from Oct 7, that is when it started”

“We must have done something bad”

That narrowed the search to one change, merged on the evening of Oct 6. It removed a React <Suspense> boundary from the root layout. We had made that change for SEO: the boundary made React move each page body into a <div hidden> that a script reveals, and we worried crawlers that don’t run JavaScript would see an empty page.

2. “Not knowing is not an acceptable answer”

A day after the “fix”, the bill was still high. The agent had a confident story, and the story was incomplete. Then it was basically saying “something is happening, I don’t know what it could be”. As problem solvers we solve problems, we don’t give up so easily. That kinda ticked me off.

“I see the oct 10 it is still high, something is still happening, take a deep look, dont make any assumptions, get a fable agent on it … Something is happening and we must figure out what it is, not knowing is not an acceptable answer”

I’m not the only one who has run into this:

“people have been trying to pass root cause analyses off as true and they sound confident but have holes like a good Swiss cheese.”

— tharkun__ on Hacker News

And before anyone touched code:

“have him report only for now, lets understand the issue before taking action”

The investigation agent came back with evidence, not a theory. It built the site just before and just after the Suspense change and compared the responses.

• Before: pages outside the prebuilt set were rendered per request and never stored.
• After: every one of those ~958,000 long-tail pages became a full static page, about 480–585 KB, written to the cache on its first visit per deployment.
• Who was visiting: OpenAI’s GPTBot, from a single IP, walked about 26,600 unique game URLs a day. A firewall rule we had written to “allow AI crawl of game content” exempted it from our crawler rate limit. 94% of the URLs it fetched weren’t even in the sitemap; it was following internal links.

Total traffic hadn’t gone up; function invocations were actually lower. What changed was how much each crawler request wrote.

GPTBot isn’t small. Vercel measured 569 million GPTBot requests across its network in one month (December 2024), and found that “none of the major AI crawlers currently render JavaScript.”

“Combine inconsiderate crawler (AI or otherwise) with a server-side logic that doesn’t really need to be there and you have a recipe for a crash, a big hosting bill, or both.”

— senko on Hacker News

3. Was the Suspense change ever needed for SEO?

The obvious fix was to put the Suspense boundary back. But that only makes sense if the boundary was never really an SEO problem. So I asked:

“can we also do some research of the suspense boundary, there is a chance that was never a problem and it was done just “to be sure""

And, honestly:

“I just want to be sure it wont affect SEO, I was never sure before we did the change and I naively thought “it will be fine""

That question turned up two things.

• External evidence: Google renders pages with JavaScript. Vercel and MERJ’s analysis of over 100,000 Googlebot fetches on nextjs.org (July 2024) found that “Streamed content via RSCs was also fully rendered.”
• Internal evidence: for the nine weeks the hidden-div markup was live, Google indexed and ranked those pages anyway. From Sep 1–16, 72 game pages earned clicks, and one ranked at an average position of 9.2 with its whole body inside <div hidden>.

So for Google, the original change had been a precaution, not a fix. Google’s own JavaScript SEO docs say the same, with a catch that matters later: “server-side or pre-rendering is still a great idea”, because “not all bots can run JavaScript.”

But I had assumed something else too: that restoring Suspense would make pages faster. The AI corrected that. A second agent built every option and measured it, and found:

• Restoring the root boundary was the worst option for SEO. It hid the body on every page, including the top pages, and moved the title, canonical and robots tags out of <head>.
• It did not make pages faster.

4. Getting cited by AI, not just ranked by Google

While we were looking at crawlers, I asked a question that is becoming more important than classic SEO:

“is there anyway we can make sure we get cited by AI? This is becoming more and more important”

LLMs have been eating a larger and larger share of search, and Google has been dropping the ball with the experience of search. For a web app like mine, where people want recommendations of games, it is very likely they would be asking an LLM “I played X and loved it, what are other games like it?”

That changed how we treated bots. The AI separated training crawlers (GPTBot, ClaudeBot, CCBot), which only collect content for training, from search and user fetchers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User), which are what get you into answers.

The bot owners’ own docs back this split. OpenAI says disallowing GPTBot means content “should not be used in training”, while sites that block OAI-SearchBot “will not be shown in ChatGPT search answers”. Perplexity says PerplexityBot “is not used to crawl content for AI foundation models”, and Anthropic runs three bots: ClaudeBot (training), Claude-SearchBot (search) and Claude-User (fetches pages for a user’s question). ChatGPT-User and Claude-User fetch pages when someone asks; they don’t build an index.

Training crawlers also give little back. In July 2025, Cloudflare measured about 1,091 pages crawled by OpenAI for every visitor it referred, and about 38,000 for Anthropic.

Then the AI checked the live firewall and found that Perplexity had been completely blocked. Perplexity crawls from Amazon’s network, and an old rule challenged that whole network. A bot can’t pass a challenge. I made the call:

“yeah make the change, use the CLI, lets allow perplexity”

5. The decision only I could make: speed over cost

The agent proposed “option D”: stop storing long-tail pages at all and render them on every request. It was cheaper per visit, and on paper it looked tidy. It depended on picking which pages were worth caching. That’s where domain knowledge mattered:

“remember we have 960K pages and most of them are game details and games-like pages, these are the SEO pages built from ~120K games in the catalog and we will rank for games randomly on SERP so any type of “lets predict the pages to build” is a failed plan”

The AI never considered the shape of the fully rendered web app, and the way SEO works for IndieDex, where each one of the ~120K games generates 8 pages that are indexable. It was operating as if this was a small site with a small catalog, thinking it could add static rendering to “some” pages and add them to the sitemap. It never considered that the games people search for are mostly random, that there would be new games releasing every week, or that we can’t know which games we are going to rank for in the SERP.

In GSC we see different games getting popular at different times. A game that just released might be popular for a few weeks, then get completely forgotten. A popular streamer may stream an old game and make it popular again for a short window.

Then I pushed on the economics instead of accepting “it’s fixed”:

“doesn’t that mean we dont really fix anything and that eventually with traffic I would see the same costs creeping up? What would that traffic have to look like that match what I am seeing today”

The answer reframed the problem. With static, CDN-cached pages, a write happens only on the first visit to a page between cache resets. Every repeat visit is free. To spend today’s $3.35 a day on real people, the site would need about 28,000 distinct pages visited per day, or about 46,000 once pages got lighter. That’s roughly 1,000× our current Google traffic, at which point the cost is a rounding error against revenue. So I chose:

“I think we go with CDN then, I’d rather the app be speedy than cheaper”

Speed for an app like this is essential. It is a discovery app: you want each page to load in under 1 second. If the user is clicking on a game they want to see and is waiting too long, it will greatly impact the UX. A bad UX makes for a bad product and bad engagement, which to me means bad for SEO, and if it is bad for SEO it eventually means I would get much less traffic. Option D was cheaper, but it was also going to make the app much slower without the CDN.

The bug was never “ISR costs money”. It was that bots that bring no value were paying the price meant for real visitors, and a weekly job was resetting the whole cache for no reason.

6. Knowing when “good enough” is right

I asked the agent to fold in one more leak it had found: the weekly data pipeline marked all ~960K pages stale every Sunday, even when a game hadn’t changed. Then I asked for an independent review:

“include the Weekly catalog reset to this plan, then proceed with the plan and at the end, summon a fable agent to review and verify the effects”

The reviewer flagged that crawlers would now see stale user reviews for longer. The AI checked and found that people see live reviews either way, because the client re-fetches them. I made the product call:

“no review refresh is needed, as long as users see the reviews and crawlers eventually see them it is fine”

What we shipped

• AI crawlers get an uncached, fully visible render. All of them, search and training alike, were added to Next.js’s htmlLimitedBots list. They get the full body inline and metadata in <head>, and nothing is written to the cache. People still get fast CDN-cached pages.
• Training crawlers are rate-limited, and citation bots are allowed, including Perplexity, which was unblocked.
• Each game’s pages are refreshed only when that game changes. A typical week now marks about 33,000 pages stale instead of 960,000. The lookup that finds changed games went from 4.0 s to 0.14 s once its columns were indexed.
• Catalog pages are lighter. They ship only the translations they use: 114 KB down to 38 KB, and each page is about 27% smaller.
• Guard rails. The build now fails if a game page picks up a catalog-wide cache tag again. A test fails if a page is missing a translation it needs.

htmlLimitedBots (added in Next.js 15.2) makes the listed bots get “blocking metadata instead of streaming metadata”, so the title and canonical land in <head>. The default list covers Google’s crawlers, Bingbot and social previews, but no AI crawlers, and setting your own list replaces it. Our list keeps the defaults.

On production after deploy, a long-tail page fetched as GPTBot comes back x-vercel-cache: BYPASS with no caching, its heading visible and its metadata in <head>. GPTBot’s pass-through dropped from about 26,600 requests a day to about 6,000, and those requests no longer write to the cache.

What happened next: Perplexity showed up

Yes, the bill came down. But the changes we made to the firewall caused the Perplexity bot to crawl at a fast pace. The ISR writes went down, but the rest of the costs went up due to the increased bot activity. So I readjusted the firewall to rate limit that bot, and now I am monitoring things.

So far, the charts are looking much healthier, and I have accomplished the goal: the app runs cheaper and is more SEO friendly.

Vercel usage dashboard, Oct 12, 2026. The write-units chart counts UTC days, so its peak lands on Oct 9; the cost chart and the table above count Pacific days.

What it means for SEO (and getting cited by AI)

• Treat SEO changes as hypotheses. The change that caused this was made “to be sure”, without evidence of a real indexing problem. Search Console data from before the change would have shown Google was fine.
• Google is not the only reader anymore. AI crawlers don’t run JavaScript. If you want to be cited, your page has to arrive as complete HTML, and your firewall must not block the fetchers that do the citing. Check it: ours silently blocked Perplexity.
• Training crawlers and citation fetchers are different audiences. Throttle the first, welcome the second.
• Static pages have a cost shape. On a long-tail SEO site, the cost tracks distinct pages times cache resets. Crawl breadth, deploy frequency and catalog-wide cache invalidations matter more than traffic volume.
• “Hidden until JavaScript runs” mostly didn’t hurt Google, but it can hurt AI. Tools like Mozilla Readability drop hidden content entirely: its visibility check (_isProbablyVisible) fails any element with the hidden attribute, and the article extractor removes nodes that fail it. That’s another reason to keep complete HTML for the pages you want cited.

Why software engineers are still valuable

The AI did a lot of the work here. It read build output byte by byte, sampled thousands of log rows, read React’s own source, ran builds of five variants, wrote the fix with full test coverage, and ran an independent review. That’s days of work, done in hours.

It also:

• gave a confident first explanation that was incomplete;
• assumed a precautionary change had been necessary;
• carried a sub-agent’s mistaken claim that Googlebot was never cached;
• proposed an option that only works if you can predict which pages will rank.

None of those mistakes was exotic. They’re the mistakes a fast, capable, context-free collaborator makes.

“these tools are actually senior developers, but they are senior developers without context. … Building a highly scalable infrastructure setup is the wrong solution if the constraints are to reduce costs.”

— filipeximenes on Hacker News

What kept the work on track wasn’t writing code. It was:

• Insisting on evidence: “dont make any assumptions … not knowing is not an acceptable answer.”
• Sequencing: “lets understand the issue before taking action.”
• Questioning past decisions, including your own: “there is a chance that was never a problem.”
• Domain knowledge the model didn’t have: “we will rank for games randomly on SERP.”
• Pushing on the economics: “eventually with traffic I would see the same costs creeping up?”
• Owning the trade-offs: “I’d rather the app be speedy than cheaper.”
• Knowing when good enough is the right answer: “as long as users see the reviews and crawlers eventually see them it is fine.”

The business context isn’t in the codebase

The business context is spread far beyond the codebase: strategies, operations, third parties. The AI can get some of the context, but only the context that is available to it. That’s the same limit I wrote about in the future of software engineering: the context that matters was never written down where the model can read it.

Not everyone buys that this lasts. One Hacker News commenter’s take on every “AI can’t do X” argument:

“Now what about project management? Train Fable 7. What about product management and talking to stakeholders? Train Fable 8. What about market research and sales? Train Fable 9.”

— ekidd on Hacker News

It may ask some questions, but I don’t believe it will be able to answer them by itself. “What is better for this business?” doesn’t do much if there isn’t a clear context of what the business is and where the owner wants to take it. There will always be variables the AI can’t access.

The job is shifting from typing the solution to steering it. That means framing the question, refusing the plausible answer, knowing the business, and making the call. That’s still engineering. If anything, it rewards the engineers who understand their systems and their users most deeply, because an AI can run every experiment you ask for, but it can’t know which question to ask.