Crawling & Indexing
Every AI answer that cites a web page starts with a crawler fetching it and an index storing it. For Google’s AI features, the documented page-level requirement is being indexed and eligible for a snippet. The site must also be included in Search’s generative AI features, the default setting (Google AI Overviews). That makes this section the foundation for the rest of the wiki.
Learning path
Section titled “Learning path”- robots.txt — what crawlers may fetch, and why blocking is not the same as de-indexing.
- Configure robots.txt for AI crawlers — separate decisions for AI search and AI training crawlers.
- Canonical URL signals — which URL represents a set of duplicates.
- Set up hreflang for translated pages — tie translations together without making them duplicates.
- XML sitemaps — help discovery, and what Google ignores in sitemaps.
- Keep a page out of Google — remove a page from results without the robots.txt trap.
- Snippet controls for AI Overviews and AI Mode — limit what AI Overviews and AI Mode use, without leaving Search.
- Google-Extended — opt out of Gemini training, which does not affect Search.
- Crawl budget — which sites need to think about it (most don’t).
- Move a site to new URLs — move to new URLs or a new domain without losing what you’ve built.
- Mobile-first indexing — Google indexes the mobile version; keep it complete.
- Claim: Duplicate content gets your site penalized — why duplicate pages are filtered, not penalized.
- URL structure — crawlable, readable URLs, and parameters that multiply them.
Entries in this topic
Guides
- Configure robots.txt for AI crawlersTreat search crawlers and training crawlers as separate decisions. Blocking a search crawler such as OAI-SearchBot removes you from that product's answers; blocking a training token such as GPTBot or Google-Extended does not affect search inclusion.
- Keep a page out of GoogleTo keep a page out of Google Search, use noindex and leave the page crawlable. robots.txt alone can't do it, because Google must crawl the page to see noindex. For permanent removal, delete or protect the content; the Removals tool hides a page within a day, but only for about six months.
- Move a site to new URLsMap old URLs to new ones with permanent (301 or 308) redirects, avoid chains, and keep redirects generally at least a year. For a new domain, submit Change of Address for every verified variant of the old domain. Expect ranking fluctuation for a few weeks or more; 301s don't lose PageRank.
- Set up hreflang for translated pagesAdd an hreflang annotation on each language version listing every version, including itself. Annotations must be reciprocal, use valid language (and optional region) codes, and point at canonical, indexable URLs.
Reference
- Canonical URL signalsRedirects, rel="canonical" and sitemap inclusion all tell Google which duplicate URL you prefer. Google treats them as signals and picks the canonical itself. Redirects and rel="canonical" are both strong signals, with redirects listed first; sitemaps are weak.
- Crawl budgetMost sites can ignore crawl budget, which is set by crawl capacity limit and crawl demand. Google's guide targets very large or fast-changing sites: 1 million+ pages changing weekly, or 10,000+ changing daily. Other sites only need an up-to-date sitemap and a regular look at the Page Indexing report.
- Google-ExtendedGoogle-Extended is a robots.txt product token, not a separate crawler. It controls whether content Google crawls may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. Google states it does not affect inclusion or ranking in Google Search, which includes AI Overviews and AI Mode.
- Mobile-first indexingGoogle indexes and ranks the mobile version of a page, crawled with its smartphone agent. Responsive design avoids mismatches because both versions share the same HTML. With dynamic serving or separate mobile URLs, the mobile version needs the same content, headings, robots meta tags, structured data and metadata as desktop, without hiding primary content behind interaction.
- robots.txtrobots.txt tells crawlers which URL paths they may fetch, and is standardised as RFC 9309. It controls crawling, not indexing. A disallowed URL can still be indexed without its content if other pages link to it.
- Snippet controls for AI Overviews and AI ModeGoogle's robots meta rules apply to AI Overviews and AI Mode. nosnippet stops a page's content being a direct input, max-snippet limits how much is used, data-nosnippet excludes parts of a page and noindex removes the page from Search. To leave AI features but keep classic results, use Search Console's site-wide Search generative AI control.
- URL structureGoogle requires URLs that follow IETF STD 66, don't change content via fragments, and use = and & for parameters. It recommends descriptive words in the audience's language, hyphens over underscores, few parameters and one case, since URLs are case sensitive. Parameter combinations such as additive filters can create floods of near-duplicate URLs that waste crawling.
- XML sitemapsA sitemap helps Google discover URLs, but does not guarantee crawling or indexing; well-linked sites may not need one. Each file is limited to 50 MB uncompressed or 50,000 URLs. Google ignores priority and changefreq, and uses lastmod only when it is consistently and verifiably accurate.