JavaScript sites (Next.js, Vue and others): making sure AI crawlers can read your content

This guide is for developers and technical leads whose sites are built with React, Next.js, Vue, Nuxt or similar frameworks. It explains why an AI crawler may receive an empty page, which rendering choices fix that, and how to check the result. Auditing a few templates takes an hour. Switching rendering mode on an existing app can take days to weeks.

The short version

1. Assume AI crawlers read only the HTML your server returns.
2. Find out what that HTML contains for each template.
3. Render the main content on the server: SSR, static generation or pre-rendering.
4. Put links, metadata and structured data in that HTML too.
5. Keep what bots receive the same as what users see.
6. Verify with no-JavaScript fetches and crawler logs, and re-check after each release.

1. Assume crawlers read only the server HTML

Google renders JavaScript. Most AI crawlers, on the best available evidence, don’t.

Google processes JavaScript pages in three phases: crawling, rendering and indexing. Pages wait in a queue until “once Google’s resources allow, a headless Chromium renders the page and executes the JavaScript”.

Officially documented · Google Search Central

Observed in 2024-12, more than 9 months ago. Answer engines change often; re-test before relying on this.

In tests published in December 2024, Vercel and MERJ found that none of the major AI crawlers they examined rendered JavaScript. They named crawlers from OpenAI, Anthropic, Meta, ByteDance and Perplexity. Gemini, using Googlebot’s infrastructure, did render JavaScript, as did AppleBot.

Observed in practice · observed 2024-12 · Vercel & MERJ

Google still recommends server-side rendering or pre-rendering because “it makes your website faster for users and crawlers, and not all bots can run JavaScript.”

Officially documented · Google Search Central

The study is one observation, now almost two years old, and none of the AI companies document their rendering. The safe design doesn’t depend on who renders: put the content in the HTML. Background: JavaScript rendering and AI crawlers.

Design for the crawler that runs no JavaScript. Every crawler that does is then a bonus, not a dependency.

Based on the Vercel and MERJ test of December 2024; AI companies do not document whether their crawlers render JavaScript.

2. Check what each template returns

Don’t audit the whole site. Pick one URL per template: home, article, product, category, docs page.

curl -s -A "OAI-SearchBot" https://example.com/page/ > raw.html
grep -c "a sentence from the middle of the main text" raw.html
grep -iE '<title>|rel="canonical"|name="description"|application/ld\+json' raw.html

For each template, record whether the raw HTML has:

Item	In raw HTML?
Main text and headings
Internal links as <a href>
Title, description, canonical
Structured data
Content behind tabs, accordions, “load more”

Then compare with Google’s view in URL Inspection: URL Inspection tool. A gap between the two is what AI crawlers miss.

3. Render the main content on the server

Modern frameworks already support server rendering. Often the fix is to stop opting out of it.

In the Next.js App Router, layouts and pages are Server Components by default. On the server, Client Components and the Server Component payload are used to prerender HTML, which the browser shows before JavaScript hydrates the page.

Officially documented · Next.js (Vercel)

Vue’s SSR guide lists better SEO as an advantage, since crawlers see the fully rendered page. It warns that if an app starts with a loading spinner and fetches content via Ajax, “the crawler will not wait for you to finish”. It recommends static generation (SSG) when only a few pages need SEO and the data is known at build time.

Officially documented · Vue.js

Choosing a mode:

Content	Mode
Marketing pages, docs, blog	Static generation at build time
Catalogue or content that changes often	SSR, or static with revalidation
Logged-in app screens	Client rendering is fine. Crawlers don’t need them

In Next.js, watch for data fetched in useEffect and for "use client" placed high in the tree. In Vue or Nuxt, check that content-critical pages aren’t client-only routes.

Google says: “Dynamic rendering is a workaround and not a recommended solution.” It recommends server-side rendering, static rendering or hydration instead.

Officially documented · Google Search Central

So serving a pre-rendered snapshot only to bots is a stopgap for legacy apps. Plan to replace it.

4. Put links and metadata in the HTML too

Crawlers find pages through links and understand them through metadata. Both fail silently when injected late.

• Use real <a href> links for navigation, not click handlers on divs: Link best practices.
• Render <title>, meta description, canonical and hreflang on the server: Canonical URL signals, Set up hreflang for translated pages.
• Output JSON-LD in the server HTML, not from a tag manager: Structured data and AI search.
• Return real status codes. A client-side “not found” screen with a 200 status is a soft 404: HTTP status codes and crawlers.
• Keep the XML sitemap generated from the same routes: XML sitemaps.

5. Serve bots the same facts as users

Microsoft’s Copilot shopping guide says: “Ensure rendered DOM contains the same facts consumers see — never serve different HTML to bots.”

Officially documented · Microsoft Advertising

Server rendering serves the same HTML to everyone, so this comes for free. Bot-only snapshots and user-agent switches are where facts drift: a price, a stock state, a date. If you must keep a snapshot service, compare its output with the live page on a schedule.

Server rendering gives every reader, human or bot, the same page. That is the whole point.

6. Verify, then keep verifying

To appear as a supporting link in AI Overviews or AI Mode, a page “must be indexed and eligible to be shown in Google Search with a snippet”.

Officially documented · Google Search Central

• Check key templates in URL Inspection and confirm they’re indexed.
• Add the curl checks from step 2 to CI, so a release that moves content client-side fails the build.
• Re-run the template audit after framework upgrades and redesigns.

What not to do

• Don’t trust a browser test. Your browser runs JavaScript. A crawler may not (step 1).
• Don’t build new sites on dynamic rendering. Google calls it a workaround (step 3).
• Don’t hide key content behind interactions that fetch it on click (step 2).
• Don’t rely on llms.txt or Markdown copies instead of fixing rendering. They aren’t a documented substitute: Claim: llms.txt improves AI search visibility.
• Don’t block crawlers from the JS and CSS that Googlebot needs to render. Google renders with them: robots.txt.

How you’ll know it’s working

Google’s crawlers can be verified with a reverse DNS lookup followed by a forward lookup, or against Google’s published IP lists.

Officially documented · Google

• Raw HTML. Every key template passes the step 2 checklist, in CI.
• Crawler visits. Verified AI search crawlers fetch your pages and get 200: Verify AI crawler traffic in server logs.
• Indexing. Pages indexed in Search Console’s page indexing report: Page indexing report.
• Citations. Fixed prompt panels and AI reports, as in Measuring AI visibility by sampling prompts and Bing AI Performance report.

Evidence at a glance

Recommendation	Evidence
Googlebot renders JS, in a queue	Official (Google)
Major AI crawlers don’t render JS	Observed (Vercel and MERJ, 2024-12)
Server-render or pre-render main content	Official (Google, Vue, Next.js)
Avoid dynamic rendering for new builds	Official (Google)
Same facts for bots and users	Official (Microsoft, for Copilot shopping)
Verify with raw fetches and logs	Official (Google)