An AI training opt out that takes one line of robots.txt arrived on October 30, backed by both Google and Microsoft on the same day. It is a real improvement on the per-provider mess it replaces, and it does not control the thing most site owners think it controls.
The two posts landed hours apart. Google said on its Search Central blog that it plans to honour a shared policy token in robots.txt, __usage_ai-training, developed in the AIPREF working group at the Internet Engineering Task Force, with implementation targeted for 2027. John Mueller explained the mechanics, and Barry Schwartz wrote it up the same day. Microsoft went considerably further in its own announcement, retiring NoArchive and NoCache as AI preferences and replacing them with two separate controls, with hard dates attached.
The industry read that as one story: a single line you can add to robots.txt to opt out of AI. It is not one story, and the single line is not the lever that matters. Model training and answer grounding are different uses of your content, they are now configured in different files, and only one of them has anything to do with whether an engine names you.
Get this wrong in the cautious direction and you quietly remove yourself from AI answers while believing you protected your copyright. Get it wrong in the relaxed direction and you publish a token that stops nothing you cared about. Both mistakes are easy to make this quarter, because the announcements use the word AI for two uses of content that have almost opposite commercial consequences.
What Did Google And Bing Announce On October 30?
Google and Microsoft both announced support for a shared robots.txt token that lets a site state one AI training preference for every provider that honours it, instead of maintaining a separate rule for each company's crawler. Microsoft additionally replaced its legacy caching directives with purpose-built preferences.
Google's half is narrow on purpose. The company said it plans to implement the shared token in 2027, and its crawler documentation points at early 2027. The token covers the AIPREF definition of AI training and nothing else, with discussions on the rest of that vocabulary still open. Mueller was explicit that existing company-specific controls keep working alongside the joint signal, so Google-Extended does not go away, and that Google's separate Search Console mechanism still governs its generative search features, which remains its own decision with its own price tag.
Microsoft's half is the operationally expensive one. It defines two preferences rather than one, gives them different scopes, sets four dates, and tells you plainly what the second one costs. It also closes a loophole that matters for anyone running legacy directives: once Microsoft detects either new preference on a site, it disregards NoArchive and NoCache on that site entirely. A half-migrated configuration is therefore not a safe middle state.
One more detail deserves attention, because it is the sort of thing that turns a tidy plan into a missed deadline. Neither preference takes effect when you publish it. It takes effect when the page is next recrawled. Microsoft notes that IndexNow may improve the odds of a recrawl without guaranteeing one, which means the real deadline for a large site is the one its crawl cycle imposes, not the one on the calendar.
What Is The Difference Between AI Training And AI Grounding?
AI Training covers using an asset to modify the learned parameters of a generative model, while AI Grounding covers using content as input to help inform an AI-generated response. Microsoft's post states both definitions directly, and adds that indexing and ranking pages in traditional search results is not grounding.
That third clause is the one to read twice. Grounding is retrieval at answer time. It is the mechanism that pulls your page into a response, it is upstream of every citation metric anyone reports, and it runs on the same earned sources engines already prefer to pull from. Training is what happens to a model months before anyone asks it a question, and your page's contribution to it is diffuse, unattributable and commercially close to worthless to you. The two uses have been filed under one acronym for three years, and the practical consequence of mixing them up is now a configuration error with a date on it.
| CONTROL | WHERE YOU SET IT | WHAT IT STOPS | WHAT IT COSTS YOU |
|---|---|---|---|
| Shared AI training token | robots.txt, site or path level | Use of the page to train generative models at every provider honouring the token | Nothing in search results or AI answers, on the providers' own account |
| Google-Extended | robots.txt, Google only | Google's model training, and Gemini Apps grounding when disallowed | Eligibility to appear in Gemini Apps responses, and the clicks that come with them |
| Bing AI Training | robots.txt or page-level tag, page wins | Use of the page to modify model parameters | Nothing in traditional Bing search results |
| Bing AI Grounding | page-level meta tag or X-Robots-Tag only | Use of the page as input to an AI-generated answer | Microsoft's own warning: significantly reduced brand visibility in AI experiences |
| Bing AI Grounding, DisallowExtended | page level only | Grounding by third parties Microsoft supplies, while its own experiences keep using the page | Visibility inside the AI products that license Bing grounding rather than crawl you |
| Search Console generative AI opt out | Search Console, a separate mechanism | Google's Search generative AI features | Presence in the AI surfaces that now sit on top of most commercial queries |
Read the first and fourth rows against each other. The control that is trivial to deploy, site-wide, in a file every team already edits, costs you nothing you were monetising. The control that would genuinely change your exposure can only be set page by page, and Microsoft attaches an explicit visibility warning to it. That asymmetry is not an accident of drafting. It is the engines telling you which door they would prefer you walk through.
“Training is a copyright conversation. Grounding is a traffic conversation. The companies that lose the next two years will be the ones who answered the second question with a policy written for the first.”
Why A Robots.txt AI Training Opt Out Is Not Enough
A robots.txt AI training opt out cannot express a grounding preference, so a site that publishes only the shared token has stated nothing about the use of its content that produces citations, and Bing will keep grounding on its pages by default.
Defaults do the work here, and they point one way. Both Bing preferences default to Allow when nothing applicable is published. So a site that does nothing stays fully available for grounding, which for most commercial publishers is the correct outcome reached by accident. A site that carefully adds the training token also stays fully available for grounding, having changed nothing about its citation exposure. Only a site that edits templates to emit page-level tags has actually touched the lever.
There is a second trap in Google's half, and it runs the other way. Mueller said that a site using the shared token to opt out of training, without explicitly specifying Google-Extended, will be treated as if it had disallowed Google-Extended. Google-Extended governs Gemini Apps grounding as well as training. So the one-line training opt out, which costs nothing at Bing, quietly removes Gemini Apps grounding eligibility at Google, and with it the clicks Mueller noted that grounding sends. One token, two providers, two different blast radii.
The reason this matters commercially is that AI answers are no longer a slice of the result page. They are the result page on the queries with budget attached. Semrush tracked 400 US commercial keywords across banking, software and ecommerce for a year and found AI Overviews on 88 percent of them in August 2026, against 45.75 percent on the same panel in September 2025. Ahrefs put AI Overviews on 83 percent of branded searches, up from roughly 61 percent in July. DemandSphere logged more than 80 percent of its tracked branded queries by late September, peaking at 90.48 percent on September 27.
Share of tracked queries showing an AI Overview, by dataset. Fixed panels, US results, appearance only, no click data.
Treat those as four vendor panels rather than a census, because that is what they are: fixed keyword sets, US results, appearance measured and clicks not. The spread between them is the honest finding. Every panel moved the same direction at roughly the same speed, which is enough to settle the only question that bears on a grounding decision. Whatever share you assume, the AI answer is now the default presentation on commercial intent, and opting out of grounding means opting out of the presentation layer your buyers see first.
What Are The Deadlines For Bing's New Controls?
Microsoft set four dates between December 2026 and September 2027, and only one of them is a true deadline: after March 1, 2027, NoArchive and NoCache stop being read as training or grounding preferences, and pages without a published preference default to Allow.
For most sites this is a calendar item rather than a project, because the correct configuration is close to doing nothing. For publishers, marketplaces and anyone currently running NoArchive across a template, it is a real migration with a crawl-rate dependency, and the work is template work rather than robots.txt work. Worth noting too that Microsoft says the Bing search syndication API is unaffected by the grounding preference, so licensing routes into third-party products are governed by DisallowExtended rather than by the ordinary grounding switch.
How Should You Set Your AI Training Opt Out?
Most commercial sites should disallow training through the shared token only if a legal or licensing position requires it, leave grounding on Allow everywhere, and accept the Gemini Apps trade-off at Google or name Google-Extended explicitly to avoid it. Publishers negotiating licensing deals are the real exception.
The logic is not complicated once the two uses are separated. Training gives you no attribution, no referral and no measurable return, so blocking it costs little and is a defensible default for a brand that cares about how its material is used. Grounding gives you the citation, the mention and whatever clicks follow, which is the entire mechanism behind getting named in a ChatGPT answer in the first place. Turning it off to make a point about copyright is paying with the asset to protest the invoice.
The one genuine complication is Google's coupling. Because the shared token without an explicit Google-Extended rule is read as disallowing Google-Extended, and Google-Extended covers Gemini Apps grounding, a blanket training opt out does cost you grounding in Gemini Apps. If that surface matters to you, say so in the file: publish the shared token for the providers you want to block and an explicit Google-Extended Allow for the one you do not. That is more verbose than the single line everyone is sharing, and it is the configuration that matches most commercial intentions.
Keep some perspective on the standard itself. Nate Hake, who took part in the IETF work, said publicly that the preference signal is not actually done at the IETF level and that significant areas of dispute remain, including his objection that the current draft largely exempts the preferences from applying to anything a user does. Google's own post calls the work in progress. So this is a direction of travel with two large implementers behind it, not a settled rule, and the sensible posture is to configure for it while expecting the vocabulary to move.
There is also the older problem underneath all of it, which no token solves. A preference signal only binds crawlers that identify themselves honestly and read the file, which is why Gary Illyes spent the same week pointing out that AI has made bot impersonation worse. Server-side verification of the crawlers hitting your pages remains the only enforcement you actually own, and it is the same argument we made about building reporting on signals an engine can switch off without telling you.
See where you are cited today
A free snapshot audit of your rankings and AI citations before we ever talk.
Josh leads work at the intersection of SEO and generative engines at Something Inc., helping B2B brands get ranked and cited across every major AI engine.