GEO best practices for technical and SEO leads

This guide is for the person who owns crawlability: a technical SEO, a platform engineer or a web lead. It turns the six-step AI search access audit into a programme you can own. Expect two or three days for the first full pass and a few hours a month afterwards. You need Search Console, Bing Webmaster Tools, CDN or WAF settings, and raw access logs.

The short version

1. Confirm key pages are indexed and eligible for snippets in Google and Bing.
2. Set robots.txt per crawler token, separating search from training.
3. Check that the CDN, WAF and hosting let the same crawlers through.
4. Remove unintended snippet limits and AI exclusion settings.
5. Make sure the main content is in the server HTML.
6. Verify real crawler visits in the logs, including Chinese search engines if that market matters.
7. Automate these checks and alert on regressions.

Each layer can block the next. Work top to bottom.

1. Indexing and snippet eligibility

Every AI search product here sits on top of a search index. If a page is not indexed, nothing downstream matters.

To appear as a supporting link in AI Overviews or AI Mode, a page “must be indexed and eligible to be shown in Google Search with a snippet”. There are no additional technical requirements.

Officially documented · Google Search Central

• Use URL Inspection on a sample of templates, not only the home page (URL Inspection tool).
• Review the Page indexing report for whole templates dropping out (Page indexing report, Diagnose a page not indexed by Google).
• Do the same in Bing Webmaster Tools, where the AI Performance report also lives (Bing AI Performance report).

2. robots.txt per token

Each AI company now runs several bots with different jobs. Your policy should say which purposes you allow, then map each purpose to tokens.

OpenAI states that the OAI-SearchBot and GPTBot settings are independent. A site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Changes take about 24 hours to be reflected in search.

Officially documented · OpenAI

Anthropic separates ClaudeBot (training), Claude-User (user-requested fetches) and Claude-SearchBot (search indexing). Blocking the latter two may reduce visibility in Claude.

Officially documented · Anthropic

For Google, AI features are part of Search, so robots.txt rules for Googlebot are the crawl control. Google-Extended applies to some of Google’s other AI systems, not to Search.

Officially documented · Google Search Central

An August 2026 audit of robots.txt on 130 company sites found that companies making an explicit choice almost never blocked search crawlers. The authors call blocking search and training crawlers together the most common and most damaging mistake.

Observed in practice · observed 2026-08 · DATA17 AI Research Lab · 解数智能 (unpublished report, not publicly available; readers can't check it)

Purpose	Tokens to review
Search indexing	Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot
User-triggered fetch	ChatGPT-User, Claude-User, Perplexity-User
Training	GPTBot, ClaudeBot, Google-Extended
Chinese search	Baiduspider, 360Spider, Sogou web spider, Bytespider

Full syntax per token: Configure robots.txt for AI crawlers and robots.txt.

Write the policy by purpose first, then by token. “Block AI” is not a policy.

3. CDN, WAF and hosting

robots.txt says what you allow. The edge decides what actually gets through. Bot-management products now ship AI-specific defaults, and those defaults change.

Google lists “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure” among the practices that still matter for AI features.

Officially documented · Google Search Central

Cloudflare’s legacy “Block AI bots” setting blocks verified bots classified as crawling for AI training, and excludes mixed-purpose bots used for both training and search. Cloudflare says that from September 15, 2026, new domains get updated defaults: training and agent bots blocked on pages with ads, search bots allowed, and mixed search-and-training crawlers blocked under any AI-training option.

Officially documented · Cloudflare

• Export the bot rules, managed rulesets and rate limits that apply to the site.
• For each search token in section 2, confirm it is allowed, not challenged.
• Check firewall logs for 403, 429 and challenge responses to verified crawler IPs.
• Re-check after any CDN plan change, new zone or provider announcement. See HTTP status codes and crawlers.

4. Snippet controls and AI exclusion

Templates, plugins and legal requests add snippet restrictions. They can quietly remove a page from AI answers while it still ranks in classic results.

nosnippet prevents content from being used as a direct input for AI Overviews and AI Mode, and max-snippet limits how much may be used.

Officially documented · Google Search Central

If you exclude a site with Search Console’s Search generative AI control, its links and content won’t appear in AI Overviews, AI Mode and related features. The control does not affect AI training (use Google-Extended) and is not a way to leave Search (use noindex).

Officially documented · Google Search Console Help

• Grep HTML and headers for noindex, nosnippet, max-snippet, data-nosnippet and X-Robots-Tag.
• Check Settings › Search generative AI on every Search Console property, including parent domains.
• Keep a list of pages where a restriction is deliberate, with the reason.

Details: Snippet controls for AI Overviews and AI Mode, Search generative AI control.

5. Rendering

Google renders JavaScript, but most AI crawlers do not. Content that appears only after client-side rendering may be invisible to them.

Google recommends server-side or pre-rendering because “not all bots can run JavaScript”.

Officially documented · Google Search Central

Observed in 2024-12, more than 9 months ago. Answer engines change often; re-test before relying on this.

In tests published in December 2024, Vercel and MERJ found that none of the major AI crawlers they examined rendered JavaScript. They named OpenAI, Anthropic, Meta, ByteDance and Perplexity crawlers. Gemini, which uses Googlebot’s infrastructure, did render JavaScript.

Observed in practice · observed 2024-12 · Vercel & MERJ

Fetch each template with JavaScript off and diff the main text against the rendered page. Fix by template, not by URL. See JavaScript rendering and AI crawlers and JavaScript sites (Next.js, Vue and others): making sure AI crawlers can read your content.

6. Verify visits in the logs

User-agent strings are easy to fake. Count only visits you can verify, by DNS or by published IP ranges.

Google’s crawlers can be verified with a reverse DNS lookup followed by a forward lookup, or against Google’s published IP lists.

Officially documented · Google

OpenAI publishes the IP addresses for each crawler as JSON: openai.com/searchbot.json (OAI-SearchBot), openai.com/gptbot.json (GPTBot) and openai.com/chatgpt-user.json (ChatGPT-User).

Officially documented · OpenAI

For the Chinese market, each engine documents its own method.

Baidu recommends a reverse DNS lookup on the IP. The hostname of Baiduspider is *.baidu.com or *.baidu.jp; Baidu calls any other hostname fake.

Officially documented · Baidu

360 Search says its crawler does not yet support nslookup checks; match the IP against the ranges it publishes instead. Its user agent ends in 360Spider.

Officially documented · 360 Search

Toutiao Search names its crawler Bytespider and listed six /24 IP ranges in November 2019.

Officially documented · Toutiao Search

Build a daily job that tags each log line as verified, unverified or spoofed per token. Method and scripts: Verify AI crawler traffic in server logs. Platform context for China: GEO in China: Doubao, Yuanbao, Qwen, ERNIE, DeepSeek and Kimi.

An allowed crawler with zero verified visits is a bug report, not a quiet week.

7. Monitoring and regression

Most access failures are regressions: a new WAF rule, a template change, a robots.txt edit in a release. The checks above should run on their own. This section is our recommended setup, not platform documentation.

• robots.txt diff: fetch it daily and alert on any change to rules for search tokens.
• Header and meta test: in CI, fail a build that adds noindex or nosnippet to an indexable template.
• No-JS content test: in CI, fetch key templates without JavaScript and assert that a known sentence is present.
• Edge test: request key URLs from outside your network with each search user agent and alert on 403, 429 or a challenge page. A spoofed UA test does not prove real crawlers pass, so pair it with log verification.
• Log dashboard: verified visits per token per day, with an alert when a token drops to zero.
• Owner and runbook: name one owner for crawler access, and document who can change CDN bot settings.

What not to do

• Don’t block search and training crawlers with one rule. It is the most common costly mistake (section 2).
• Don’t trust robots.txt alone. The CDN can override it (section 3).
• Don’t count unverified user agents. They inflate crawler traffic (Verify AI crawler traffic in server logs).
• Don’t block AI bots by IP when robots.txt would do. Anthropic notes IP blocks stop its crawlers from reading robots.txt (Verify AI crawler traffic in server logs).
• Don’t treat a pass as a promise of citations. These are preconditions (Google Search Essentials).
• Don’t build special files for AI to rank. llms.txt is ignored by Google Search (Claim: llms.txt improves AI search visibility).

How you’ll know it’s working

• All checks green for every key template, recorded with a date (AI search access audit).
• Verified visits from each allowed search token, steady or rising.
• Pages appearing in Search Console’s generative AI report (Generative AI performance report: AI Overviews and AI Mode in Search Console) and Bing’s AI Performance report.

Microsoft: “AI Performance does not measure rankings, authority, performance, or importance. It simply shows which content was cited.”

Officially documented · Microsoft Bing

• Citation share in a fixed prompt panel, reported as a trend (Measuring AI visibility by sampling prompts, For marketing leads: measuring GEO and reporting it upward).

Evidence at a glance

Recommendation	Evidence
Indexed and snippet-eligible	Official (Google)
robots.txt by purpose and token	Official (OpenAI, Anthropic, Google); observed common mistake (DATA17, 2026)
CDN and WAF allow search crawlers	Official (Google, Cloudflare)
No unintended snippet limits or AI exclusion	Official (Google)
Server-rendered main content	Official (Google) and observed (Vercel/MERJ, 2024)
Verify crawlers by DNS or IP list	Official (Google, OpenAI, Baidu, 360, Toutiao)
Automated regression tests	Hypothesis (our recommended practice)