AI search access audit
Short answer
Before any content work, confirm AI search systems can reach a page. Run six documented checks: indexing and snippet eligibility, robots.txt, CDN and firewall, snippet restrictions, server-rendered content, and verified crawler visits in the logs.
Before you start
Section titled “Before you start”- Pick a handful of representative URLs (a key guide, a product page, a hub) rather than the whole site.
- You need access to Google Search Console, the live
robots.txt, CDN or firewall settings, and server or CDN logs.
1. Indexed and eligible for a snippet
Section titled “1. Indexed and eligible for a snippet”Check with Search Console’s URL Inspection tool. Not indexed means stop here and fix indexing first.
2. Search crawlers allowed in robots.txt
Section titled “2. Search crawlers allowed in robots.txt”Each AI search product uses its own crawler token, and blocking it removes you from that product. Confirm the tokens you care about are allowed: Googlebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot. Details per token: Configure robots.txt for AI crawlers.
3. Not blocked by CDN, firewall or hosting
Section titled “3. Not blocked by CDN, firewall or hosting”Bot-protection rules often return 403 or a challenge page to crawlers that robots.txt allows. Check firewall logs for blocked requests from verified crawler IPs (step 6).
4. No unintended exclusions or snippet restrictions
Section titled “4. No unintended exclusions or snippet restrictions”In Search Console, check Settings › Search generative AI. If the site is set to exclude, it won’t appear in AI Overviews or AI Mode at all (Search generative AI control). Child properties can inherit that setting from a parent.
Then search the HTML and response headers of each URL for noindex, nosnippet, max-snippet and data-nosnippet. Templates and plugins add these more often than people expect. See Snippet controls for AI Overviews and AI Mode.
curl -sI https://example.com/page/ | grep -i x-robots-tagcurl -s https://example.com/page/ | grep -iE 'name="(robots|googlebot)"|data-nosnippet'5. Main content in the server HTML
Section titled “5. Main content in the server HTML”Fetch each URL without JavaScript and confirm the main text, headings and links are there. Background: JavaScript rendering and AI crawlers.
6. Crawlers actually visit
Section titled “6. Crawlers actually visit”Count only verified visits (Verify AI crawler traffic in server logs). If a search crawler you allow has no verified visits, something earlier in this list is probably still blocking it.
Record the result
Section titled “Record the result”| Check | Pass / fail | Notes |
|---|---|---|
| 1. Indexed, snippet-eligible | ||
| 2. robots.txt allows search crawlers | ||
| 3. CDN / firewall allows them | ||
| 4. No AI exclusion or unintended snippet rules | ||
| 5. Content in server HTML | ||
| 6. Verified crawler visits |
Common mistakes
Section titled “Common mistakes”- Starting with content changes while a crawler is blocked at the firewall.
- Reading “allowed in robots.txt” as “can access”. Steps 3 and 6 exist because the two often differ.
- Treating a pass as a promise of citations. These checks are preconditions. Google states that meeting requirements does not guarantee crawling, indexing or serving.