SEO Recap covering November 1, 2026: Shared robots.txt AI training opt-out line

Daily summary of what matters in SEO, GEO, AEO, and AI search generated with Claude Code (beware of hallucinations)

---

Technical SEO

Google, Microsoft, and OpenAI back a shared robots.txt AI training opt-out
Source: Search Engine Journal

• The 3 companies said on Friday they will honor a shared robots.txt line so a site can state one AI training preference to every participating provider instead of writing per-crawler rules. None of the 3 supports it yet.
• The line comes from an IETF Internet Draft by Anthropic's Kevin Kelley, filed as an individual draft (not official IETF standardization). It adds no new directive, using a policy token __usage_ai-training with ordinary Allow and Disallow rules.
• Existing robots.txt rules do not count toward the shared preference. A wildcard group (User-Agent: *) can never match the token, a file without the group signals no preference, and the token does not control crawling.
• The companies plan different implementations:
  • Google will treat a shared opt-out like a Google-Extended block when Google-Extended is unspecified, targeting early 2027, covering Gemini training and grounding in Gemini Apps and Vertex AI but not Google Search inclusion.
  • Microsoft splits training from a separate page-level AI Grounding preference, honoring new options February 1, 2027 and dropping NoArchive and NoCache as training or grounding signals after March 1, 2027.
  • OpenAI aims to honor the signal in the next few months while its current robots.txt policy stays in effect, and Anthropic says support across its services will take time.
• Microsoft is the only one replacing controls sites already use. Sites relying on NoArchive or NoCache to stay out of grounding must record a temporary preference in Bing Webmaster Tools, which opens December 15 and is limited to sites still using those tags that day.