Robots.txt Generator
Runs in your browser — your data is never uploaded.
Build a robots.txt file for your site — the paths you want left alone, the sitemap line every crawler looks for, and an optional block for the AI training crawlers. The output explains the two things this file is most often expected to do and cannot: a blocked page is not removed from Google, and a blocked page cannot be read, so a noindex tag on it is never seen.
How to use Robots.txt Generator
- 1
List the paths you want crawlers to skip — /admin, /cart, /search.
- 2
Add your sitemap URL, and tick the AI crawler block if you want one.
- 3
Click “Generate”, then save the result as robots.txt in your site root.
Share
Embed this tool on your site
Paste this where you want the tool to appear. It runs entirely in your visitor's browser — no uploads, no account, no tracking.
Please keep the attribution line — it's what keeps these tools free.
Need a different size, a dark theme, or a different tool? Build an embed lets you preview it first.
Frequently asked questions
Does robots.txt remove a page from Google?+
No, and this is the single most expensive misunderstanding about the file. Disallow stops the page being crawled. A URL that other sites link to can still be indexed from those links alone, showing in results with no description. To remove a page you have to let it be crawled and serve a noindex directive — blocking it guarantees the noindex is never read.
What is the difference between "Disallow:" and "Disallow: /"?+
One character, and the entire site. An empty Disallow means nothing is blocked. Disallow: / blocks everything. They sit next to each other in every example on the web, and a staging file copied to production is the usual way a site disappears from search overnight.
Can I block ChatGPT, Claude and other AI crawlers?+
You can ask them to stay away, and the major ones do honour robots.txt. Tick the option and the file adds a group for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and the rest. It is a request, not a wall — the file is a convention, and nothing enforces it on a crawler that chooses to ignore it.
Where does the file go?+
The root of each host: https://example.com/robots.txt. A crawler looks nowhere else, and never applies one host’s file to another, so blog.example.com needs its own. A file at /public/robots.txt does nothing at all.
Are the paths case-sensitive?+
Yes, and they match by prefix. Disallow: /Admin does not block /admin, while Disallow: /admin also blocks /administrator and /admin-guide. Anchor a rule with $ when you mean exactly that path and nothing beginning with it.