Free Robots.txt Generator

The robots.txt generator writes the small text file that tells crawlers which parts of your site they may visit. Choose a base rule, list the paths to block and the exceptions to allow, add your sitemap URL, and optionally block AI training crawlers. Copy the result and save it as robots.txt in your site root.

Key takeaways

  • Robots.txt controls crawling, not indexing. To keep a page out of Google, use noindex instead.
  • One wrong line, such as Disallow: /, can remove a whole site from search. Always double-check.
  • Blocking AI crawlers is a business choice: it protects content but may reduce visibility in AI answers.

How to create your file

  1. Keep the base rule on Allow all search engines for a live site.
  2. List folders that have no search value, one per line, such as /wp-admin/, /cart/ or internal search results.
  3. Add exceptions that must stay open, such as /wp-admin/admin-ajax.php on WordPress. The preset button fills both for you.
  4. Paste your sitemap URL so crawlers find all important pages.
  5. Save the output as robots.txt at https://yourdomain.com/robots.txt.

Crawling is not indexing

This is the most misunderstood point in technical SEO. Disallow stops well-behaved bots from fetching a URL, but if other sites link to it, Google can still index the bare URL without its content. To remove a page from results, let it be crawled and add a noindex meta tag, which you can build with the meta tags generator. Blocking a page in robots.txt and adding noindex at the same time does not work, because the crawler never sees the noindex.

What to block and what to leave open

Usually blockNever block
Admin and login areasCSS and JavaScript files
Cart, checkout and account pagesImages you want in image search
Internal search and endless filter URLsPages you want to rank
Staging copies (block the whole host)Your sitemap

Google renders pages like a browser. If it cannot load your CSS and scripts, it may misjudge the layout and mobile-friendliness of every page.

Crawl budget and duplicate URLs

Large shops and directories often produce thousands of near-identical URLs through sorting, filters and session parameters. Crawlers waste time on them, and the real pages get crawled less often. Blocking those patterns in robots.txt, for example Disallow: /*?orderby=, saves crawl budget. For duplicates that still need to pass link equity, use canonical tags instead, so the signals flow to the main version rather than disappearing.

AI crawlers and GEO

Bots such as GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot collect content for AI models and answer engines. Blocking them protects your content from being used for training, but it can also reduce the chance that AI assistants quote and link to you. For most businesses that want to be recommended in AI answers, we suggest leaving answer-engine bots open and deciding case by case on pure training bots. Our GEO service helps you choose based on where your customers search.

Test before and after

Before uploading, read the file line by line: each User-agent group applies only to the bots it names, and the most specific matching rule wins. After uploading, open yourdomain.com/robots.txt in a browser to confirm it is live, then check the robots.txt report in Google Search Console for fetch errors. Re-test after migrations, theme changes and when a staging site goes live, because staging rules copied to production are one of the most common causes of sudden traffic loss we see. Run the on-page SEO checker on a few key pages to confirm they are still indexable.

A safe starting file for WordPress

For most WordPress sites, a short file is the best file: allow everything, block /wp-admin/ while allowing admin-ajax.php, and list the sitemap your SEO plugin creates, such as /sitemap_index.xml. Do not block /wp-content/ or /wp-includes/, because they hold the CSS, scripts and images Google needs to render your pages. Add more rules only when you have a clear reason, such as a filter parameter that creates thousands of URLs. A shorter file is easier to audit and much harder to break by accident.

Need more than a free tool?

Our team runs full audits, fixes and growth plans for sites in 10 languages.
Explore our SEO services

Frequently asked questions

Where do I upload robots.txt?

In the root of the domain, so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file.

Does WordPress already have a robots.txt?

WordPress creates a virtual one. Uploading a real file, or editing it through Rank Math or Yoast, replaces it.

Can robots.txt hide private pages?

No. The file is public and only a request to polite bots. Protect private content with a password or login.

Should I block AI bots?

It depends on your goals. Blocking protects content from training; allowing answer engines can bring visibility and visits from AI search.

How fast does Google read changes?

Google usually caches robots.txt for up to 24 hours. You can request a refresh from the Search Console report.

How we built this tool: the checks and thresholds follow Google Search Central documentation, Schema.org and the IETF/W3C standards, and we tune them against what we see on client sites every month. The tool runs in your browser and does not store the text or URLs you enter. Questions or a bug? Tell us.

1

A free month of SEO for your website

We offer a free month of SEO that covers everything: technical analysis, competitor analysis, keyword research, your business goals, SEO fundamentals, on-page SEO and AI search optimization (GEO). All the work is done by a dedicated team.

Start my free month
Free consultation via WhatsApp
تم التصميم والسيو بواسطة قروش السيوواتساب +20 112 057 7167أعلن معنا