One robots.txt line now means two different things to Google and Microsoft.

The shared robots.txt opt-out from AI training reads like a privacy setting. Google plans to read it as a Gemini Apps visibility decision as well. Microsoft does not.

· 8 minute read

A business owner who adds one line to robots.txt to stop AI training on their content thinks they have made a privacy decision. On October 30, 2026, Google and Microsoft each published how they plan to read that line, and their readings do not match. At Google, the line also takes the site out of Gemini Apps answers. At Microsoft, it declines training and leaves the site citable. The business writes one line, and two engines reach two different conclusions about whether it can be cited.

The shared AI training opt-out is one robots.txt group addressed to every participating provider

The shared AI training opt-out is a robots.txt group under the token __usage_ai-training. It tells participating AI providers not to train on a site's content. Google says the policy token is based on work from the AIPREF working group at the Internet Engineering Task Force, and describes it as a way to set broad preferences across participating AI providers. Microsoft's Bing team describes the same robots.txt line as the way to set its AI Training preference for a whole site or for one path.

Search Engine Journal reports that Google, Microsoft and OpenAI plan to honor the line. OpenAI's reading cannot be compared here because it is not documented: Search Engine Journal reports that neither OpenAI's blog post nor its crawler documentation says what happens when the shared line and a GPTBot rule conflict.

At Google, opting out of AI training also ends Gemini Apps grounding eligibility

Opting out of AI training through the shared token takes a site out of Gemini Apps grounding, in three documented steps. First, Google's October 30 post says that when a website uses the policy token to opt out of training, and Google-Extended is not explicitly specified, Google plans to interpret that as if the publisher had disallowed Google-Extended. Second, Google's crawler documentation says Google-Extended controls both training of future Gemini models and grounding: supplying content from the Google Search index to the model at prompt time, in Gemini Apps and in Grounding with Google Search on Vertex AI. Third, the same post says that websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. A site that has disallowed it is not.

No Google setting separates the two uses. Declining training while remaining citable in Gemini Apps is not an option Google offers.

The size of that loss can be checked today. Google's post promises clicks via grounding in Gemini Apps and gives no figure for how many. A business can open the referral sources in its analytics, find the row for Gemini, and read the visits it sent over the last 90 days. That number, not a guess, is its stake.

Two robots.txt files show Google and Microsoft reaching different conclusions from the same line

Microsoft's Bing announcement, also dated October 30, 2026, treats AI training and AI grounding as independent preferences. Training is set by the shared robots.txt line. Grounding is a separate page-level setting that defaults to Allow when nothing is published. Google-Extended, by contrast, covers both uses at once. Here are two short files and what each engine concludes from them.

File one, the shared group alone, in the form Bing's post shows:

``` User-Agent: __usage_ai-training Disallow: / ```

Google concludes: no Gemini training, and no Gemini Apps grounding, because the file is read as a Google-Extended disallow. Microsoft concludes: no training, and grounding at Allow.

File two, the shared group plus a named Google-Extended rule:

``` User-Agent: __usage_ai-training Disallow: / User-agent: Google-Extended Allow: / ```

Google concludes: Gemini training permitted, and the site eligible for Gemini Apps grounding. Microsoft concludes: no training, and grounding at Allow.

Microsoft gives the same answer for both files. Google's answer flips on one named group, and the price of the flip is permitting Gemini training.

The shared token reaches exactly the publishers who never named Google-Extended

Google-Extended already ties Gemini training to grounding, so it is fair to ask what the shared token changes. A publisher who disallows Google-Extended has always given up grounding along with training. On that reading, the shared token only extends an existing bundle.

What the shared token changes is the audience. Google-Extended is a Google-specific control that a publisher had to look up and name. The shared token is designed for the person who will never look up a vendor's token, and that is exactly the person least likely to know that at Google the training answer is also the grounding answer.

Google's rule is worded to reach that person and no one else. The interpretation applies only when Google-Extended is not explicitly specified. A file that names Google's token, with Allow or Disallow, carries its author's own answer. The interpretation decides the outcome only for the author who never wrote Google-Extended, and that author is the shared token's intended user.

The token's name does not warn that author. It reads __usage_ai-training. Nothing in the line says grounding, and Bing's post calls it the AI Training preference. A control built for people who will not read vendor documentation should not carry a consequence that only vendor documentation reveals.

Google Search and AI Overviews are untouched, and that is what hides the cost

The strongest case against treating the shared AI training opt-out as a visibility decision is that it overstates the stakes. Google's crawler documentation states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. AI Overviews and AI Mode appear inside Google Search, so on that documentation a site using the shared opt-out stays eligible for the AI answers shown in Search. Two more points support the case. Nothing is live: Search Engine Journal reports that none of Google, Microsoft or OpenAI supports the shared line yet, with Google aiming for 2027 and Microsoft's preferences taking effect on February 1, 2027. And for many businesses, withholding content from model training is worth more than a citation in one assistant.

The Search carve-out narrows the claim, and the narrowed claim stands. The shared line costs a site Gemini Apps and Vertex AI grounding, not AI Overviews. That carve-out is also what hides the cost: rankings, Search traffic and AI Overviews appearances stay where they were, so every report the business watches says the line did nothing. The loss shows up only in Gemini's answers and in the Gemini referral row, which no rank tracker reads.

The delay is the reason to decide now. Google's interpretation will apply to whatever the file says when support arrives, and a line added before then as a privacy step will be read then as a grounding decision. Google says it plans this reading, and the plan could change before 2027. A file that names Google-Extended explicitly means the same thing whichever way Google finalizes it.

The trade can be a good one. A business may decide that Gemini Apps citations are worth giving up. That decision should be made on purpose, as a visibility decision, and not discovered later.

A business that wants to be cited should name Google-Extended in robots.txt before 2027

The position here is plain: a business that wants to be cited should not publish the shared __usage_ai-training group without a Google-Extended rule beside it. If Gemini Apps citations matter more than withholding content from Gemini training, add a group with "User-agent: Google-Extended" and "Allow: /". If withholding training matters more, write "Disallow: /" under Google-Extended by name and accept the loss of Gemini Apps grounding. Either file records a decision. The shared group alone records a default that Google chose, not the business.

Microsoft needs nothing more in the file. Bing grounding is a page-level setting, and a page that sets nothing is grounded under the default, Allow.

At Google, the training answer is also the grounding answer, and the shared line was built for people who will never look that up.

At Google, the shared line is a grounding decision. At Microsoft, it is a training preference. The file has not become more complicated. Its meaning has.

Further reading

  1. Google Search Central Blog, John Mueller: Google's announcement that it plans to treat the shared training opt-out as a Google-Extended disallow, and that Gemini Apps grounding eligibility depends on allowing Google-Extended developers.google.com/search/blog/2026/10/ai-training-control
  2. Google crawler documentation: what Google-Extended covers (Gemini training and grounding) and the statement that it does not affect Search inclusion or ranking developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
  3. Microsoft Bing, Krishna Madhavan: Microsoft's announcement making AI training and AI grounding separate preferences, with dates and defaults blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding
  4. Search Engine Journal, Matt G. Southern: independent reporting on Google, Microsoft and OpenAI plans for the shared training line, and that none supports it yet searchenginejournal.com/google-microsoft-openai-shared-ai-training-opt-out