Is robots.txt still a thing?

Yes. Every major search engine and AI company reads robots.txt, and it became an internet standard, RFC 9309, in 2022. AI crawlers made it more important, because it is the main way to choose which AI bots may read your site.

robots.txt is a voluntary convention, not a law or a contract. Writing one is always legal, and reputable crawlers follow it. Whether a crawler that ignores it breaks the law depends on the country and the case, so ask a lawyer if that matters to you.

How do I block GPTBot?

Add a group with the line User-agent: GPTBot followed by Disallow: / to your robots.txt. This keeps your pages out of OpenAI model training. To stay in ChatGPT search, leave OAI-SearchBot allowed.

Does blocking AI crawlers hurt SEO?

Blocking AI training bots does not hurt Google rankings. Google-Extended only controls Gemini training and grounding, not Google Search. Blocking Googlebot or Bingbot does remove you from search results, and blocking AI search bots removes you from AI answers.

llms.txt is a proposed markdown file at the root of a site that summarises the site and links to its key pages for AI assistants. It is not an official standard, and no major AI company has confirmed that it uses it for ranking.

Where do I put robots.txt?

Put robots.txt at the root of each host, for example https://yourproduct.com/robots.txt. A file in a folder is ignored, and each subdomain, such as app.yourproduct.com, needs its own file.

All free tools

robots.txt generator for AI crawlers, with llms.txt

Choose which crawlers may read your site, from Googlebot to GPTBot, and copy a ready robots.txt. Test any path against the file, check the files on your live domain, and write an llms.txt that lists your key pages for AI assistants.

Presets

Crawlers and whether each one may crawl your site
Crawler	Company	Access
Search enginesBlocking these removes you from Google or Bing.
Googlebot	Google
Bingbot	Microsoft
AI searchFetch pages to cite them in AI answers.
OAI-SearchBot	OpenAI
Claude-SearchBot	Anthropic
PerplexityBot	Perplexity
AI user fetchFetch a page when a person asks the assistant to.
Perplexity-User	Perplexity
ChatGPT-User	OpenAI
Claude-User	Anthropic
AI trainingCollect pages to train future models.
Bytespider	ByteDance
CCBot	Common Crawl
GPTBot	OpenAI
ClaudeBot	Anthropic
meta-externalagent	Meta
anthropic-ai	Anthropic (old token)
Amazonbot	Amazon
Google-Extended	Google
Applebot-Extended	Apple

Sitemap URL

Disallowed paths, one per line

Every crawler you allow skips these paths. Use them for the app, the admin area and the API, not for secrets: anyone can read robots.txt. There is no crawl-delay option because Google ignores that rule.

Your robots.txt

# robots.txt made with the BlockSnap generator

# Everyone else: allowed, except private paths
User-agent: *
Disallow: /admin
Disallow: /app
Disallow: /api

# Blocked: ai training
# ByteDance
User-agent: Bytespider
# Common Crawl
User-agent: CCBot
# OpenAI
User-agent: GPTBot
# Anthropic
User-agent: ClaudeBot
# Meta
User-agent: meta-externalagent
# Anthropic (old token)
User-agent: anthropic-ai
# Amazon
User-agent: Amazonbot
# Google
User-agent: Google-Extended
# Apple
User-agent: Applebot-Extended
Disallow: /

# Allowed: search engines
# Google
User-agent: Googlebot
# Microsoft
User-agent: Bingbot
Disallow: /admin
Disallow: /app
Disallow: /api

# Allowed: ai search
# OpenAI
User-agent: OAI-SearchBot
# Anthropic
User-agent: Claude-SearchBot
# Perplexity
User-agent: PerplexityBot
Disallow: /admin
Disallow: /app
Disallow: /api

# Allowed: ai user fetch
# Perplexity
User-agent: Perplexity-User
# OpenAI
User-agent: ChatGPT-User
# Anthropic
User-agent: Claude-User
Disallow: /admin
Disallow: /app
Disallow: /api

Sitemap: https://example.com/sitemap.xml

Save it as robots.txt at the root of your domain, for example yourproduct.com/robots.txt.

robots.txt tester

Crawler

Path or URL

Blocked

Line 28: Disallow: / From the group for GPTBot.

Follows RFC 9309: the most specific user-agent group applies, the longest matching path wins and Allow wins a tie. * matches any characters and $ marks the end.

Site or product name

One-line summary

Say what the product is and who it is for. It becomes the quote line.

Details (optional)

Section: Docs

Section: Product

Section: Optional

Links here are the ones an assistant can skip when it has little room.

Your llms.txt

# Acme Analytics

> Acme Analytics is product analytics for SaaS teams: it tracks sign-ups, activation and churn without code.

Acme has a free plan for up to 10,000 events a month. Paid plans start at $49 a month. Data is stored in the EU.

## Docs

- [Quick start](https://acme.example/docs/start): Install the snippet in 5 minutes
- [API reference](https://acme.example/docs/api): REST endpoints and limits

## Product

- [Features](https://acme.example/features): Funnels, retention and alerts
- [Pricing](https://acme.example/pricing): Plans and event limits
- [Security](https://acme.example/security): SOC 2, GDPR and data location

## Optional

- [Changelog](https://acme.example/changelog)

Save it as llms.txt at the root of your domain, next to robots.txt. Use full URLs.

Check your current files

How to use it

01Choose a preset, for example Block AI training, allow AI search.
02Switch single crawlers between Allowed and Blocked if you need to.
03Add your sitemap URL and the paths no crawler should fetch, such as /app and /api.
04Test a crawler and a path in the tester, then copy or download robots.txt.
05Open the llms.txt tab, fill in your name, summary and key links, and save llms.txt next to robots.txt.

What is robots.txt?

robots.txt is a plain text file at the root of a site, such as example.com/robots.txt, that tells crawlers which paths they may fetch. It is a request, not access control: well-behaved crawlers follow it, but it does not stop anyone from opening a page.

The file holds groups. Each group starts with one or more User-agent lines that name crawlers, followed by Allow and Disallow lines with paths. A crawler follows the group that names it, or the User-agent: * group when no group names it. A Sitemap line can point crawlers to your sitemap.

How SaaS sites treat AI crawlers

About 9 in 10 SaaS sites let AI crawlers in. In our scan of 198 SaaS homepages, only 9.4% of the sites whose robots.txt answered block any AI bot, and 8.9% block training bots while leaving AI search and user fetches open. 0.5% block every crawler.

60.6% of the 198 sites serve an llms.txt file, and 94.8% of robots.txt files name a sitemap. The most blocked bot is Bytespider (ByteDance), at 6.3%. No site in the scan blocks an AI search bot such as OAI-SearchBot or PerplexityBot.

The block shares below are of the 191 sites whose robots.txt answered; the llms.txt share is of all 198 sites.

Share of SaaS sites that block each AI crawler (191 sites with a robots.txt)
Crawler	Company	Job	Sites blocking	Share
Bytespider	ByteDance	Training	12	6.3%
CCBot	Common Crawl	Training	9	4.7%
GPTBot	OpenAI	Training	6	3.1%
ClaudeBot	Anthropic	Training	6	3.1%
meta-externalagent	Meta	Training	6	3.1%
anthropic-ai	Anthropic (old token)	Training	4	2.1%
Amazonbot	Amazon	Training	4	2.1%
Google-Extended	Google	Training	3	1.6%
Applebot-Extended	Apple	Training	3	1.6%
Perplexity-User	Perplexity	User fetch	1	0.5%
OAI-SearchBot	OpenAI	AI search	0	0.0%
ChatGPT-User	OpenAI	User fetch	0	0.0%
Claude-SearchBot	Anthropic	AI search	0	0.0%
Claude-User	Anthropic	User fetch	0	0.0%
PerplexityBot	Perplexity	AI search	0	0.0%

As of 21 October 2026 · Source: BlockSnap scan of SaaS homepages (robots.txt and llms.txt at the site root)

AI crawlers by job

AI companies run separate crawlers for separate jobs, so you can block one job and keep another. The table lists the tokens to put on the User-agent line and what blocking each one does.

AI crawler tokens, their owners and what blocking them does
Token	Company	Job	What blocking it does
Bytespider	ByteDance	Training	Asks ByteDance not to collect pages for its AI models.
CCBot	Common Crawl	Training	Keeps pages out of the Common Crawl dataset, which many AI models train on.
GPTBot	OpenAI	Training	Keeps pages out of OpenAI model training. ChatGPT search is not affected.
ClaudeBot	Anthropic	Training	Keeps pages out of Anthropic model training.
meta-externalagent	Meta	Training	Asks Meta not to collect pages for its AI models.
anthropic-ai	Anthropic (old token)	Training	An older Anthropic token. Some sites still list it next to ClaudeBot.
Amazonbot	Amazon	Training	Asks Amazon not to collect pages for its services and AI models.
Google-Extended	Google	Training	Keeps pages out of Gemini training and grounding. It does not change Google Search ranking.
Applebot-Extended	Apple	Training	Keeps pages out of Apple model training. Applebot still crawls for Siri and Spotlight.
OAI-SearchBot	OpenAI	AI search	Removes pages from ChatGPT search results and their citations.
Claude-SearchBot	Anthropic	AI search	Removes pages from Claude search results.
PerplexityBot	Perplexity	AI search	Removes pages from the Perplexity search index and answers.
Perplexity-User	Perplexity	User fetch	Asks Perplexity not to open pages for users. Perplexity says this fetcher generally ignores robots.txt.
ChatGPT-User	OpenAI	User fetch	Asks ChatGPT not to open pages for users. OpenAI says robots.txt may not apply to these user requests.
Claude-User	Anthropic	User fetch	Asks Claude not to open pages when a user asks it to.

As of October 2026 · Source: Crawler documentation published by each company

Training vs search vs user fetch: which AI bots should you block?

Block training bots if you do not want your content in future AI models; block search bots only if you accept leaving AI answers. Blocking a training bot such as GPTBot, ClaudeBot or Google-Extended keeps your pages out of future model training, and it does not remove you from AI search.

Blocking a search bot such as OAI-SearchBot, Claude-SearchBot or PerplexityBot removes your pages from that assistant's answers and citations. For most SaaS sites that is lost traffic: people now ask assistants which tool to buy.

User-fetch agents such as ChatGPT-User, Claude-User and Perplexity-User open a page because a person asked the assistant to. They act for that person, and some companies say these fetchers may not follow robots.txt. If a page must stay private, put it behind a login.

What is llms.txt?

llms.txt is a proposed markdown file at /llms.txt that gives AI assistants a short summary of your site and links to its key pages. It starts with an H1 name and a quoted one-line summary, then lists links under H2 headings, as described at llmstxt.org.

It is not a standard, and no major AI company has confirmed that it uses llms.txt for ranking or citations. It is cheap to add and it can help tools and agents that do read it, so treat it as a small bet, not an SEO fix.

Is robots.txt legal or binding?

robots.txt is a voluntary convention, not a law and not a lock. It started in 1994 and became a standard, RFC 9309, in 2022. Search engines and the large AI companies say they follow it, but nothing in the file stops a crawler that ignores it.

Use authentication for private content, because a Disallow line tells everyone where that content is. Whether ignoring robots.txt has legal effect depends on the country, the site's terms and what the crawler does with the data. This page is not legal advice.

Questions

Is robots.txt still a thing?: Yes. Every major search engine and AI company reads robots.txt, and it became an internet standard, RFC 9309, in 2022. AI crawlers made it more important, because it is the main way to choose which AI bots may read your site.
Is robots.txt legal?: robots.txt is a voluntary convention, not a law or a contract. Writing one is always legal, and reputable crawlers follow it. Whether a crawler that ignores it breaks the law depends on the country and the case, so ask a lawyer if that matters to you.
How do I block GPTBot?: Add a group with the line User-agent: GPTBot followed by Disallow: / to your robots.txt. This keeps your pages out of OpenAI model training. To stay in ChatGPT search, leave OAI-SearchBot allowed.
Does blocking AI crawlers hurt SEO?: Blocking AI training bots does not hurt Google rankings. Google-Extended only controls Gemini training and grounding, not Google Search. Blocking Googlebot or Bingbot does remove you from search results, and blocking AI search bots removes you from AI answers.
What is llms.txt?: llms.txt is a proposed markdown file at the root of a site that summarises the site and links to its key pages for AI assistants. It is not an official standard, and no major AI company has confirmed that it uses it for ranking.
Where do I put robots.txt?: Put robots.txt at the root of each host, for example https://yourproduct.com/robots.txt. A file in a folder is ignored, and each subdomain, such as app.yourproduct.com, needs its own file.

BlockSnap sites ship with robots.txt, a sitemap and llms.txt

BlockSnap builds your SaaS landing page from proven sections and publishes it with robots.txt, a sitemap and llms.txt already in place.

Get early access