Verify AI crawler traffic in server logs

Before you start

• Access to raw server or CDN logs with the client IP address and user-agent of each request.
• A shell with host (or dig) for DNS lookups, or a script that can fetch JSON and match IPs against CIDR ranges.

1. Google: reverse, then forward DNS

Run a reverse DNS lookup on the IP address from your logs. Check that the domain is googlebot.com, google.com or googleusercontent.com. Then run a forward DNS lookup on that domain name and check that it resolves to the same IP address. For large-scale checks, Google recommends matching IPs against its published IP lists instead.

Officially documented · Google

host 66.249.66.1                              # → crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com          # → 66.249.66.1  (must match)

Google publishes the lists as JSON files per crawler category (common crawlers, special-case crawlers, user-triggered fetchers), linked from its verification page.

2. OpenAI and Perplexity: published IP ranges

OpenAI publishes the IP addresses for each crawler as JSON: openai.com/searchbot.json (OAI-SearchBot), openai.com/gptbot.json (GPTBot) and openai.com/chatgpt-user.json (ChatGPT-User).

Officially documented · OpenAI

Perplexity publishes www.perplexity.com/perplexitybot.json for PerplexityBot and www.perplexity.com/perplexity-user.json for Perplexity-User.

Officially documented · Perplexity

Each file lists CIDR prefixes. A request is verified when its user-agent names the crawler and its IP falls inside one of that crawler’s prefixes.

3. Anthropic

Anthropic’s crawler help article links a list of IP addresses its crawlers use. A source IP on that list indicates the crawler is coming from Anthropic. It also warns that blocking by IP “may not work correctly or persistently guarantee an opt-out”. IP blocks stop its crawlers from reading robots.txt, so use robots.txt to opt out.

Officially documented · Anthropic

4. Baidu: reverse DNS

Baidu recommends a reverse DNS lookup (host, nslookup or dig -x on the IP). The hostname of Baiduspider is *.baidu.com or *.baidu.jp; Baidu calls any other hostname fake.

Officially documented · Baidu

host 123.125.66.120   # → Baiduspider-123-125-66-120.crawl.baidu.com

Baidu’s FAQ stops at the reverse lookup. Adding the forward lookup used for Google costs nothing and also catches a spoofed PTR record.

5. 360 and Toutiao: published IP ranges

360 Search says its crawler does not yet support nslookup checks; match the IP against the ranges it publishes instead. Its user agent ends in 360Spider.

Officially documented · 360 Search

Toutiao Search names its crawler Bytespider and listed six /24 IP ranges in November 2019.

Officially documented · Toutiao Search

That list is from 2019 and has no machine-readable version, so treat a mismatch as “unverified”, not as proof of a fake.

Verify

• Spot-check: pick a request from each claimed crawler and run the check by hand.
• In reports, separate claimed from verified crawler traffic. Unverified requests with a crawler’s user-agent are a different population and shouldn’t be counted as that crawler.

Rollback

Verification is read-only. If you added firewall rules based on it, remove them and fall back to robots.txt rules (Configure robots.txt for AI crawlers).

Common mistakes

• Trusting the user-agent string alone.
• Caching IP lists forever. The files carry a creationTime and change over time; refresh them on a schedule.
• Blocking unverified “Googlebot” traffic by user-agent string. It blocks nothing once the impostor changes its string. Meanwhile, IP-based blocking of real crawlers can stop them from reading your robots.txt.