Technical Implementation
Content only counts if crawlers receive it. These entries cover what happens between a request and the HTML a crawler gets back. Does JavaScript run? Is a request that claims to be a crawler really one?
Start here
Section titled “Start here”- JavaScript rendering and AI crawlers — what Googlebot renders, and what a 2024 study found most AI crawlers do not.
- Verify AI crawler traffic in server logs — confirm a request is really from Google, OpenAI, Anthropic or Perplexity.
- HTTP status codes and crawlers — how 4xx, 5xx and redirects change what gets indexed.
- Page experience and Core Web Vitals — Core Web Vitals and how much they matter.
- AI agents and your website — how browser agents read a site, and what helps them.
Entries in this topic
Concepts
Guides
Reference
- HTTP status codes and crawlers4xx URLs (except 429) are not indexed, and 401 or 403 should not be used to limit crawl rate. 5xx and 429 slow crawling, and indexed URLs are kept for a while before being dropped. A 2xx makes content eligible but doesn't guarantee indexing; 301 is a strong redirect signal, 302 a weak one.
- JavaScript rendering and AI crawlersContent that only appears after client-side rendering may be invisible to most AI crawlers. A December 2024 Vercel and MERJ study found OpenAI's, Anthropic's and Perplexity's crawlers did not execute JavaScript. Googlebot does render it with headless Chromium, in a queue after crawling.
- Page experience and Core Web VitalsCore Web Vitals are among the signals Google's ranking systems use, but there is no single page experience signal. Good thresholds are LCP within 2.5 s, INP under 200 ms and CLS under 0.1. Good scores don't guarantee top rankings, and relevance can outweigh a sub-par page experience.