robots.txt

Location and format

• Served at /robots.txt at the root of each host and protocol (https://example.com/robots.txt does not cover https://shop.example.com/).
• Plain text, UTF-8. Groups start with one or more user-agent lines followed by allow / disallow rules.

Supported fields (Google)

Field	Purpose
user-agent	Which crawler the group applies to; * matches any crawler without a more specific group.
disallow	Path prefix the crawler should not fetch.
allow	Path prefix that may be fetched, overriding a broader disallow.
sitemap	Absolute URL of a sitemap. Not tied to any group.

Google does not support other fields such as crawl-delay. In case of conflicting rules, including wildcards, Google uses the least restrictive rule.

Officially documented · Google

* (any sequence) and $ (end of URL) are supported in paths.

Limits and caching

Google enforces a 500 KiB size limit and ignores content after it. It generally caches robots.txt for up to 24 hours, and may cache longer when refreshing fails (for example on timeouts or 5xx errors).

Officially documented · Google

Crawling is not indexing

A page blocked by robots.txt can still appear in results, without a description, if other pages link to it. To keep a page out of results, allow crawling and use noindex instead.

Officially documented · Google Search Central

noindex inside robots.txt is not supported. Use the robots meta tag or X-Robots-Tag header instead; Keep a page out of Google has the steps.

Baidu

Baiduspider follows robots.txt, and rules can target its product-specific user agents such as Baiduspider-image. Baiduspider-video does not support the rules. Blocking Baiduspider removes the site from Baidu search results “and in any other the search results which is provided by Baidu”. Baiduspider-cpro and Baiduspider-ads work under customer agreements and do not follow robots.txt.

Officially documented · Baidu

The FAQ names no separate token for Baidu’s AI products, so it documents no way to treat them differently from Baidu search. Google, by contrast, has a separate token for Gemini (Google-Extended).

Other Chinese search engines

360 Search supports user-agent, allow and disallow, with tokens 360Spider, 360Spider-Image and 360Spider-Video. It adds its own Indexpage field to mark frequently updated pages.

Officially documented · 360 Search

Sogou’s crawler supports robots.txt, but a changed file may take several weeks to take effect, and pages you block cannot be found in Sogou search.

Officially documented · Sogou

Toutiao Search’s crawler introduction (2019) gives the Bytespider user agent and IP ranges but does not mention robots.txt.

Officially documented · Toutiao Search

Example

user-agent: *
disallow: /search
allow: /search/about

sitemap: https://example.com/sitemap-index.xml

See also

• Configure robots.txt for AI crawlers — per-crawler rules for AI search and training bots.