# robots.txt rules for AI agents

> A file that says which crawlers and agents may fetch which pages.

Source: https://agentrelations.org/standards/robots-txt

## What it is

robots.txt sits at the root of a website and lists which automated visitors may fetch which paths. AI providers publish the names their crawlers use. Some names belong to training crawlers. Others belong to agents that fetch a page because a person just asked about it.

An analogy

A sign on the door that says who may come in and which rooms they may enter.

## At a glance

Owner

IETF (RFC 9309); the agent names belong to each AI provider

Maturity

Stable RFC 9309, published 2022. Each provider publishes its own user-agent names (GPTBot, ClaudeBot, Google-Extended and the agents that fetch on behalf of a person).

Where it lives

[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)

Framework domain

[Discover: Be found](https://agentrelations.org/framework/domains/discover)

What it serves

- Control which agents can reach your site
- Separate training crawlers from agents working for a person

## Who should care

The roles this standard touches most. See all the roles on the personas page.

- **Engineering and product.** Build the access, data, endpoints and actions that agents use, on open standards.
- **Marketing and brand.** Own the story agents tell: accurate facts, positioning, reputation and presence where agents look.
- **Legal, risk and security.** Set the policies: which agents may do what, identity, data, disclosure and liability.

[Framework personas →](https://agentrelations.org/framework/personas)

## How to adopt it

1. 1
List the agents you know of and decide for each: allow, block or limit.
1. 2
Allow the agents that fetch for a person, since blocking them hides you from the answer.
1. 3
Treat training crawlers as a separate policy choice.
1. 4
Review the file every quarter, because new agent names appear often.

## The Institute’s view

Adopt

This is the one agent standard every organization already has to get right. Many sites block everything by accident and then wonder why agents never mention them. Decide on purpose.

Standards change fast. Last reviewed November 2026.

[← Back to the Standards Map](https://agentrelations.org/standards)
