Skip to content
GEO Casa Wiki
GEO Casa Wiki
Crawling & IndexingChange logObserved in practiceOutside ChinaReviewed

Crawler documentation changes

Short answer

A dated record of changes to official crawler documentation from Google, OpenAI and the IETF, found by this wiki's weekly source checks. Each change says whether the substance we cite changed or only the location.

This log records changes to the crawler documentation that entries in this wiki cite. Changes are found by npm run check:sources, which runs weekly. Every change is re-read before it is logged, to tell a page that moved from a policy that changed.

Changes

Dates are when we observed and verified the change, not necessarily when the platform made it.

  1. Google's crawl budget and HTTP status code guides moved to the crawling documentation site

    Observed in practice

    The crawl budget guide (previously search/docs/crawling-indexing/large-site-managing-crawl-budget) and the HTTP status codes guide (previously search/docs/crawling-indexing/http-network-errors) now live under developers.google.com/crawling/docs/; old URLs redirect. Re-checked the facts this wiki cites, such as the 1 million+ and 10,000+ page thresholds and 4xx/5xx handling.

    Google: Crawl budget management · Google: How HTTP status codes affect Google's crawlers

  2. Google's crawler verification guide moved to the crawling documentation site

    Observed in practice

    "Verifying Googlebot and other Google crawlers" moved from developers.google.com/search/docs/crawling-indexing/verifying-googlebot to developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests; the old URL redirects. Re-checked: verification is still a reverse DNS lookup (googlebot.com, google.com or googleusercontent.com) followed by a forward lookup, or matching against Google's published IP lists, now split into per-category JSON files.

    Google: Verify requests from Google crawlers and fetchers

  3. Google's robots.txt specification moved to the crawling documentation site

    Observed in practice

    "How Google interprets the robots.txt specification" moved from developers.google.com/search/docs/… to developers.google.com/crawling/docs/robots-txt/robots-txt-spec; the old URL now returns 404 without a redirect. Re-checked: the 500 KiB limit, the least-restrictive rule for conflicts, unsupported crawl-delay and up-to-24-hour caching are unchanged.

    Google: How Google interprets the robots.txt specification

  4. OpenAI's crawler overview moved to developers.openai.com

    Observed in practice

    The overview of OpenAI crawlers moved from platform.openai.com/docs/bots to developers.openai.com/api/docs/bots; the old URL redirects. Re-checked that OAI-SearchBot and GPTBot are still independent settings and that robots.txt changes take about 24 hours to reflect in search.

    OpenAI: Overview of OpenAI crawlers

  5. RFC 9309 short link now opens the RFC's metadata page

    Observed in practice

    rfc-editor.org/rfc/rfc9309 now redirects to the info page rather than the text. The specification itself is unchanged; this wiki now links the HTML text directly.

    IETF: RFC 9309: Robots Exclusion Protocol

  6. Google's crawler list lives on the crawling documentation site

    Observed in practice

    The list of Google's common crawlers, including the Google-Extended token, is served from developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers; the former Search Central path redirects there.

    Google: Google's common crawlers

Sources

  1. How Google interprets the robots.txt specificationGoogle · official · ✓ 2026-10-25
  2. Overview of OpenAI crawlersOpenAI · official · ✓ 2026-10-25
  3. Google's common crawlersGoogle · official · ✓ 2026-10-24
  4. Verify requests from Google crawlers and fetchersGoogle · official · ✓ 2026-10-25
  5. Crawl budget managementGoogle · official · ✓ 2026-10-25
  6. How HTTP status codes affect Google's crawlersGoogle · official · ✓ 2026-10-25
  7. RFC 9309: Robots Exclusion ProtocolIETF · standard · ✓ 2026-10-25