Learning paths: Developers
- Who it is for
- You build or run a site and want search engines and AI crawlers to reach and read it.
- What you get
- A site AI crawlers can access, render and cite, and logs that prove it.
Reading order
- AI search access auditStart with an audit of what blocks AI search today.
- Configure robots.txt for AI crawlersWhich user agents to allow, and what each one is for.
- Google-ExtendedThe token that controls Gemini training, not Search.
- JavaScript rendering and AI crawlersWhich AI crawlers run JavaScript and which see only HTML.
- HTTP status codes and crawlersHow status codes change crawling and indexing.
- Verify AI crawler traffic in server logsCheck in server logs that the crawlers are real and arriving.
- Snippet controls for AI Overviews and AI Modenosnippet and friends: what they do to AI Overviews.
- Structured data and AI searchWhat structured data does and does not do for AI features.
- XML sitemaps vs IndexNowTelling engines about new URLs: sitemaps vs IndexNow.
- AI agents and your websiteAgents that browse and act on your site.