OpenAI backs a shared robots.txt signal for AI training OpenAI backs a shared robots.txt signal for AI training

One purpose, several providers

OpenAI describes a voluntary effort to express the same training preference to multiple participating providers. Its announcement commits to implementing support in the coming months and publishing details when that happens. For a web team, the immediate task is therefore planning a configuration change. Removing existing bot rules now would discard controls that OpenAI says it will continue to respect. A shared entry should eventually simplify maintenance, especially for publishers that manage many domains with the same content policy.

The proposal separates use from retrieval

The linked draft puts a purpose-specific token into familiar robots.txt groups. It uses path-based Allow and Disallow rules to express preferences about training use. The policy group does not itself tell a crawler to stop fetching a page. That distinction lets an operator express a content-use preference separately from its access rules. It also creates a testing requirement: parsing a file successfully does not demonstrate that a downstream training process applies the preference. Operators need to decide which evidence they will accept for each part of that chain.

Existing controls already split the jobs

OpenAI documents GPTBot for training and OAI-SearchBot for search. A publisher can restrict the former while permitting the latter. Anthropic makes a similar split with ClaudeBot and Claude-SearchBot, and also documents Claude-User for user-requested retrieval. Anthropic says disabling its user-directed crawler prevents that retrieval and can reduce visibility in those requests. These controls give site owners separate choices, at the price of maintaining several provider names. For a publisher that wants discovery without training collection, an undifferentiated block on every AI crawler would express the wrong policy.

An access boundary still needs enforcement

RFC 9309 treats robots.txt as a crawler convention and explicitly distinguishes it from protective measures such as authentication. A training preference therefore belongs in a different operational category from a password, account entitlement or paywall. The practical consequence is simple: private material should stay behind enforced access controls. A team reviewing this proposal should inventory public pages separately from customer records and paid content. That avoids asking a public preference file to carry a security responsibility it was never designed to perform.

Test the policy that readers actually want

Our suggested rollout starts with a small matrix: training, search indexing and user-requested retrieval, each with an explicit owner and decision. Keep the deployed robots.txt version with the change record, then test representative paths before changing every domain. Include public articles, subscription pages and any paths with exceptions. Request logs can help check retrieval behavior, while a separate review must address providers' documented training-use commitments. Treat these as two tests with different evidence. Combining them into one green status would hide the most important distinction in the proposal. Give exceptions an expiry date and a named reviewer. Otherwise a temporary rule for one integration can survive long after that integration disappears. Include a rollback copy of the previous file. If a deployment changes search discovery unexpectedly, the operator should be able to restore the earlier configuration without reconstructing it.