Bing and Google back one robots.txt switch for AI training opt-outs

Bing and Google back one robots.txt switch for AI training opt-outs

30 October 2026: Microsoft Bing and Google have both said they will honour a shared robots.txt token, __usage_ai-training, that lets a website refuse the use of its content for training generative AI models across every provider that adopts it. Bing goes further: it is retiring the NoArchive and NoCache tags as AI controls, adding a separate page-level setting for "grounding" in AI answers, and setting dates. Sites that rely on the old tags to keep content out of Copilot and Bing's AI answers have until 1 March 2027 to switch.

Bing's dates for sites using NoArchive or NoCache

Krishna Madhavan, a principal product manager at Microsoft Bing, wrote that publishers "should not have to choose between being discoverable in Bing and deciding how generative models use their work". The schedule:

• 15 December 2026: temporary grounding preferences open in Bing Webmaster Tools, only for sites using NoArchive or NoCache on that date.
• 1 February 2027: Bing starts honouring the new training and grounding preferences as it crawls.
• 1 March 2027: NoArchive and NoCache stop being read as training or grounding preferences.
• 30 September 2027: the temporary Webmaster Tools preferences expire, or earlier for each page once it is recrawled.

Bing asks for the Webmaster Tools step because it may not recrawl a site between February and 1 March, and a preference it has not crawled does not count.

Training and grounding are now separate choices

Training means changing the learned parameters of a generative model. It can be refused for a whole site or a path in robots.txt with User-Agent: __usage_ai-training and Disallow: /, and set per page with a meta tag or an X-Robots-Tag header (ai-training:disallow), where the page setting overrides robots.txt.

Grounding means using a page as input to an AI-generated answer, in Bing, Copilot or Microsoft 365. It exists only at page level, with three values: Allow, the default; Disallow; and DisallowExtended, which keeps a page in Microsoft's own experiences, such as Copilot and AI Overview, but out of grounding services Microsoft sells to others, including Web IQ and the grounding API in Foundry. Both settings default to Allow, neither changes eligibility for ordinary Bing results, and neither sets up licensing or payment. Bing warns that refusing grounding "will significantly reduce" a site's visibility in AI answers.

Google maps the token to Google-Extended

John Mueller, a search advocate at Google, wrote that when a site opts out of training with the shared token and does not name Google-Extended, "Google plans to interpret that as if the publisher had disallowed Google-Extended". The company-specific control keeps working alongside it. Google gives no firm date: it aims to support the control "along with other AI providers in 2027", and ties the proposal to the EU AI Act's code of practice for general-purpose model providers, which treats robots.txt as a primary way to state machine-readable preferences.

The token comes from the IETF's AI Preferences working group. The current text is an individual Internet-Draft by Kevin Kelley of Anthropic, published on 30 October and not yet adopted by the IETF.

What site owners can do before February

A site with no AI preference today needs to do nothing; both companies treat no setting as an allow. A site that used NoArchive or NoCache to keep pages out of Bing's AI answers needs the new page-level grounding tag on those pages and the Webmaster Tools entry before 1 March. Our SEO software category lists 20 products.

Company profile on TrustList: Google

Company profile on TrustList: Microsoft

Related on TrustList:

• Obsero raises $1.2m seed to track how AI assistants recommend brands
• Drupal fixes critical Field UI flaw with a no-login path via add-ons
• Google and OpenAI both retire their custom assistant builders

Sources

• Microsoft Bing Blog: Your content, your choice: clearer preferences for training and grounding, 30 October 2026
• Google Search Central Blog: Supporting a shared AI training control for websites, 30 October 2026
• IETF Datatracker: draft-kelley-aipref-training-preference-signal-00, Machine-Readable Training Preference Signal, 30 October 2026