OpenAI plans support for a common website AI-training preference signal

A publisher that wants to deny use of its articles for generative AI training has usually had to learn the names of multiple crawlers, update separate rules and track which provider respects which instruction. OpenAI announced on 30 October that it intends to honour a common training-preference signal in robots.txt, joining an effort to make one expression of preference legible to several providers. Its key implementation qualification is explicit: OpenAI expects to introduce support in the coming months, not immediately.[1]

An Internet-Draft linked from that announcement provides a more precise picture of what the shared signal would mean. Rather than inventing a new robots.txt command, the proposal uses the familiar User-Agent and Disallow grammar with a purpose-specific token: __usage_ai-training. A rule such as a disallowance for a content path is intended to describe permitted use for model training, not necessarily whether a bot can visit that path.[2]

A purpose preference is not a crawler block

In the draft, a website operator can place User-Agent: __usage_ai-training in a robots.txt group and then use existing path-level Allow or Disallow instructions. A disallowed path means a preference against the described training use; an allowed path means a preference in favour. If no group includes this token, the specification says no preference has been expressed by this mechanism. That is different from treating silence as a blanket consent or refusal.[2]

The separation from ordinary crawler rules is essential. A publisher might want a service to access articles so they can appear in search and answer citations, while declining training on the same text. The draft explicitly says its policy token is not a crawling directive, and existing crawler product-token rules continue to operate independently. In practice, the publisher needs to manage both access and usage rather than assuming either control covers the other.[2]

This approach also avoids maintaining entirely separate path lists for every participating AI company. Under the draft's precedence scheme, specific proprietary preferences can still be relevant; the common token is not a mechanism for wiping out all existing provider-specific controls. OpenAI says its present crawler-control system will remain in force as the new system is introduced.[1][2]

The standard is still being made

The referenced text is an IETF Internet-Draft, not a final Internet Standard or a statute. The broader IETF AI Preferences work is developing vocabulary for uses such as generative-model training. A voluntary machine-readable statement becomes operational only to the extent that providers actually adopt, interpret and honour it. It does not by itself determine copyright ownership, licensing rights or whether content already used in a dataset can be removed.[2][3]

There is also a technical distinction between 'training' and every other possible AI use. The draft's definition centres on using an asset to modify the learned parameters of a generative AI model. It expressly excludes exclusively non-generative classification, ranking or scoring systems. A site owner seeking to govern retrieval, summarisation or other uses should not assume that a training-specific token covers them.[2][3]

What website operators can do today

Website operators still need their existing provider-specific controls while implementation is pending. A proposed common rule can express a preference, but its presence in robots.txt does not establish that every provider recognises it. The operational check is whether each provider's published implementation supports the token, how it resolves competing rules and which uses it covers.

The significance is narrower, and more useful, than a claim that publishers have suddenly gained a universal opt-out. This is a proposal for interoperability among voluntary controls. If implemented widely, it could substantially reduce the administrative work of expressing a training preference; the evidence of its effectiveness will be the providers' later documented behaviour.[1][2]

Report an error or suggest a correction