The robots.txt generator writes the small text file that tells crawlers which parts of your site they may visit. Choose a base rule, list the paths to block and the exceptions to allow, add your sitemap URL, and optionally block AI training crawlers. Copy the result and save it as robots.txt in your site root.
Key takeaways
- Robots.txt controls crawling, not indexing. To keep a page out of Google, use noindex instead.
- One wrong line, such as Disallow: /, can remove a whole site from search. Always double-check.
- Blocking AI crawlers is a business choice: it protects content but may reduce visibility in AI answers.
How to create your file
- Keep the base rule on Allow all search engines for a live site.
- List folders that have no search value, one per line, such as /wp-admin/, /cart/ or internal search results.
- Add exceptions that must stay open, such as /wp-admin/admin-ajax.php on WordPress. The preset button fills both for you.
- Paste your sitemap URL so crawlers find all important pages.
- Save the output as robots.txt at https://yourdomain.com/robots.txt.
Crawling is not indexing
This is the most misunderstood point in technical SEO. Disallow stops well-behaved bots from fetching a URL, but if other sites link to it, Google can still index the bare URL without its content. To remove a page from results, let it be crawled and add a noindex meta tag, which you can build with the meta tags generator. Blocking a page in robots.txt and adding noindex at the same time does not work, because the crawler never sees the noindex.
What to block and what to leave open
| Usually block | Never block |
|---|---|
| Admin and login areas | CSS and JavaScript files |
| Cart, checkout and account pages | Images you want in image search |
| Internal search and endless filter URLs | Pages you want to rank |
| Staging copies (block the whole host) | Your sitemap |
Google renders pages like a browser. If it cannot load your CSS and scripts, it may misjudge the layout and mobile-friendliness of every page.
Crawl budget and duplicate URLs
Large shops and directories often produce thousands of near-identical URLs through sorting, filters and session parameters. Crawlers waste time on them, and the real pages get crawled less often. Blocking those patterns in robots.txt, for example Disallow: /*?orderby=, saves crawl budget. For duplicates that still need to pass link equity, use canonical tags instead, so the signals flow to the main version rather than disappearing.
AI crawlers and GEO
Bots such as GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot collect content for AI models and answer engines. Blocking them protects your content from being used for training, but it can also reduce the chance that AI assistants quote and link to you. For most businesses that want to be recommended in AI answers, we suggest leaving answer-engine bots open and deciding case by case on pure training bots. Our GEO service helps you choose based on where your customers search.
Test before and after
Before uploading, read the file line by line: each User-agent group applies only to the bots it names, and the most specific matching rule wins. After uploading, open yourdomain.com/robots.txt in a browser to confirm it is live, then check the robots.txt report in Google Search Console for fetch errors. Re-test after migrations, theme changes and when a staging site goes live, because staging rules copied to production are one of the most common causes of sudden traffic loss we see. Run the on-page SEO checker on a few key pages to confirm they are still indexable.
A safe starting file for WordPress
For most WordPress sites, a short file is the best file: allow everything, block /wp-admin/ while allowing admin-ajax.php, and list the sitemap your SEO plugin creates, such as /sitemap_index.xml. Do not block /wp-content/ or /wp-includes/, because they hold the CSS, scripts and images Google needs to render your pages. Add more rules only when you have a clear reason, such as a filter parameter that creates thousands of URLs. A shorter file is easier to audit and much harder to break by accident.
Need more than a free tool?
Our team runs full audits, fixes and growth plans for sites in 10 languages.Frequently asked questions
Where do I upload robots.txt?
Does WordPress already have a robots.txt?
Can robots.txt hide private pages?
Should I block AI bots?
How fast does Google read changes?
How we built this tool: the checks and thresholds follow Google Search Central documentation, Schema.org and the IETF/W3C standards, and we tune them against what we see on client sites every month. The tool runs in your browser and does not store the text or URLs you enter. Questions or a bug? Tell us.