Standard · Discover
robots.txt rules for AI agents
A file that says which crawlers and agents may fetch which pages.
What it is
robots.txt sits at the root of a website and lists which automated visitors may fetch which paths. AI providers publish the names their crawlers use. Some names belong to training crawlers. Others belong to agents that fetch a page because a person just asked about it.
An analogy
At a glance
- Owner
- IETF (RFC 9309); the agent names belong to each AI provider
- Maturity
- Stable RFC 9309, published 2022. Each provider publishes its own user-agent names (GPTBot, ClaudeBot, Google-Extended and the agents that fetch on behalf of a person).
- Where it lives
- RFC 9309
- Framework domain
- Discover: Be found
- What it serves
- Control which agents can reach your site
- Separate training crawlers from agents working for a person
Who should care
The roles this standard touches most. See all the roles on the personas page.
- Engineering and product. Build the access, data, endpoints and actions that agents use, on open standards.
- Marketing and brand. Own the story agents tell: accurate facts, positioning, reputation and presence where agents look.
- Legal, risk and security. Set the policies: which agents may do what, identity, data, disclosure and liability.
How to adopt it
- 1
List the agents you know of and decide for each: allow, block or limit.
- 2
Allow the agents that fetch for a person, since blocking them hides you from the answer.
- 3
Treat training crawlers as a separate policy choice.
- 4
Review the file every quarter, because new agent names appear often.
The Institute’s view
This is the one agent standard every organization already has to get right. Many sites block everything by accident and then wonder why agents never mention them. Decide on purpose.
Standards change fast. Last reviewed November 2026.