Technical SEO8 min readنوفمبر 1, 2026

Robots.txt Optimization: Rules, Examples and Key Mistakes

Robots.txt Optimization: Rules, Examples and Key Mistakes - Robots.txt Optimization: Rules, Examples and
Robots.txt Optimization: Rules, Examples and Key Mistakes - Robots.txt Optimization: Rules, Examples and

By Islam Ramadan, SEO and GEO Specialist · Last updated: October 9, 2026

What Is Robots.txt Optimization?

Robots.txt optimization means writing the robots.txt file so crawlers skip URLs that waste their time, without blocking any content you need indexed. The file controls crawling, not indexing. It lives at the root of your domain and uses four directives: user-agent, disallow, allow and sitemap. One wrong line can hide an entire site.

DirectiveWhat it doesExample
User-agentNames the crawler a group of rules applies toUser-agent: *
DisallowBlocks crawling of a pathDisallow: /cart/
AllowPermits a path inside a blocked oneAllow: /cart/help/
SitemapGives the location of a sitemapSitemap: https://www.example.com/sitemap.xml
*Wildcard: any sequence of charactersDisallow: /*?sort=
$End of URLDisallow: /*.pdf$
#Comment# Block internal search

The rules follow Google’s guide on how to write a robots.txt file. Other search engines support the same core directives, with some differences.

What Does Robots.txt Do, and What Does It Not Do?

Google says a robots.txt file is used primarily to manage crawler traffic to your site. It tells well-behaved crawlers which URLs they may request.

It does not do three things people expect:

  • It does not keep a page out of search results. A disallowed page can still be indexed if other sites link to it, shown as a bare URL without a description.
  • It does not protect private content. The file is public, and not every crawler obeys it.
  • It does not remove pages already indexed.

To keep a page out of Google, use a noindex rule or password protection. And the page must stay crawlable for noindex to be seen.

Where Must the File Be Located?

The file must be named robots.txt and placed at the root of the host: https://www.example.com/robots.txt. It cannot sit in a subfolder.

  • One file per host.
  • Rules apply only to that protocol, host and port. A subdomain such as shop.example.com needs its own file.
  • The file must be plain text, UTF-8 encoded.
  • If no file exists, crawlers assume everything may be crawled.

What Is the Syntax?

A file is made of groups. Each group begins with one or more user-agent lines, followed by rules.

A minimal file allowing everything, with a sitemap:

  • User-agent: *
  • Disallow:
  • Sitemap: https://www.example.com/sitemap.xml

Key rules:

  • Paths are case-sensitive. /Cart/ and /cart/ are different.
  • A crawler follows the most specific group that matches its name, and ignores the others.
  • The longest matching rule wins between allow and disallow.
  • Lines that are not valid rules are ignored.
  • Each directive goes on its own line.

Which Examples Are Useful?

Block internal search results

  • User-agent: *
  • Disallow: /search
  • Disallow: /*?s=

Block cart, checkout and account pages

  • User-agent: *
  • Disallow: /cart/
  • Disallow: /checkout/
  • Disallow: /account/

Block filter and sort parameters

  • User-agent: *
  • Disallow: /*?sort=
  • Disallow: /*&sort=
  • Disallow: /*?filter=

Block one crawler from a folder

  • User-agent: Googlebot
  • Disallow: /drafts/

Block a file type

  • User-agent: *
  • Disallow: /*.pdf$

Adapt patterns to your own URLs, and test each one. Google’s guide to faceted navigation recommends robots.txt for stopping the crawling of filter URLs that do not need to be in search.

Want this handled for you?

Our SEO services start with a full audit and a written plan, followed by a monthly report that shows what changed and why.

See our SEO services

What Should You Block?

Block URLs that have no value in search and consume crawling.

Usually blockUsually leave open
Internal search resultsProduct, category and article pages
Cart, checkout, accountCSS and JavaScript files
Filter and sort combinationsImages you want in image search
Session and tracking parametersPages that carry noindex
Admin and login areasYour sitemap
Endless calendar or infinite URLsAnything you want ranked

Small sites often need to block almost nothing. A file allowing everything and pointing to the sitemap is a perfectly good robots.txt.

Which Mistakes Cause the Most Damage?

  1. “Disallow: /” left over from development. It blocks the whole site. Check after every launch.
  2. Blocking CSS or JavaScript. Google needs these to render pages. Blocking them can make pages appear broken or empty.
  3. Blocking a page to “deindex” it. The crawler can no longer see a noindex tag, and the URL may stay in results.
  4. Blocking pages that have a canonical or redirect. Those signals cannot be read either.
  5. Case errors. A rule for /Blog/ does not block /blog/.
  6. A missing trailing slash. “Disallow: /private” also blocks /private-offers.
  7. Rules in the wrong group. A crawler with its own group ignores the general “*” group.
  8. Different files on www and non-www, or on subdomains, forgotten.
  9. Blocking the sitemap or pages listed in it.
  10. Relying on crawl-delay, which Googlebot does not process.

Should You Block AI Crawlers?

It is a business decision, and the answer differs by crawler. AI-related bots fall into three kinds.

KindPurposeEffect of blocking
Training crawlersCollect content to train modelsContent is not used for training
Search crawlersBuild an index for AI search answersYou disappear from that assistant’s answers
User-triggered fetchersFetch a page when a user asksThe assistant cannot read the page on request

Examples from the companies’ own documentation:

  • Google-Extended is a token that controls whether your content is used for training Gemini models and for grounding. Google says it does not affect inclusion or ranking in Google Search.
  • OAI-SearchBot must be allowed for a site to be eligible for ChatGPT search.
  • PerplexityBot is used to surface and link sites in Perplexity’s search results, and is not used to train foundation models, according to Perplexity. Its documentation says the user-triggered Perplexity-User generally ignores robots.txt, because a person requested the page.

Blocking everything labelled “AI” removes you from AI answers as well. If you want visibility in those tools, allow their search crawlers, and decide separately about training. Our guide to how to rank in ChatGPT covers access.

How Do You Test the File?

  1. Open it in a browser, in a private window, to confirm it loads publicly.
  2. Use the robots.txt report in Search Console. It shows the files Google found, when it last fetched them, and any errors.
  3. Use the URL Inspection tool on a few important pages. It reports if a page is blocked by robots.txt.
  4. Crawl the site with a crawler set to obey robots.txt, and list what was blocked. See our guide to using an SEO crawler.
  5. Keep the file in version control, and review every change.

After editing, Google refreshes its cached copy periodically. You can request a recrawl of the file in the robots.txt report.

How Does Robots.txt Fit with Other Controls?

GoalUse
Stop crawling of low-value URLsrobots.txt
Keep a page out of resultsnoindex (page must be crawlable)
Choose between duplicatesCanonical tag
Move a page permanently301 redirect
Protect private contentPassword or login
Remove a page fastSearch Console removals, plus noindex

Do not combine robots.txt blocking with noindex on the same URL. Pick the one that matches the goal.

For large sites, see our guide to crawl budget optimization. To understand indexing statuses, see the Google index.

Frequently Asked Questions

Does robots.txt stop a page from being indexed?

No. It stops crawling, not indexing. A blocked page can still appear in results as a bare URL if other sites link to it. To keep a page out of search, allow it to be crawled and add a noindex rule, or protect it with a password.

Where must the robots.txt file be located?

At the root of the host, named exactly robots.txt, for example https://www.example.com/robots.txt. It cannot be placed in a subfolder. Each subdomain and each protocol needs its own file, because rules apply only to the exact host where the file is posted.

Should I block AI crawlers in robots.txt?

Decide per crawler. Blocking training crawlers keeps your content out of model training. Blocking search crawlers, such as OAI-SearchBot or PerplexityBot, removes you from those assistants’ answers. Google-Extended affects Gemini training and grounding, not Google Search. Allow search crawlers if you want AI visibility.

Do I need a robots.txt file at all?

Not strictly. Without one, crawlers assume the whole site may be crawled. A simple file that allows everything and lists your sitemap is good practice. Larger sites benefit from rules that keep crawlers away from search results, filters and other low-value URLs.

How quickly do robots.txt changes take effect?

Google caches the file and refreshes it periodically, generally within about a day. You can ask for a recrawl in the robots.txt report in Search Console. Pages blocked or unblocked by the change are then handled as they are next crawled.

Related Guides

Want this done for you?

See what each monthly plan includes, or send us your site and questions on WhatsApp. We do not promise rankings; we show the work and the data every month.

View SEO packages Chat on WhatsApp

Sources

  1. Google Search Central: How to write and submit a robots.txt file
  2. Google Search Central: Introduction to robots.txt
  3. Google Search Central: Block Search indexing with noindex
  4. Google Search Central: Managing crawling of faceted navigation URLs
  5. Google Search Central: Google’s common crawlers
  6. OpenAI Help Center: ChatGPT search
  7. Perplexity Docs: Perplexity crawlers
About the author: Islam Ramadan

Islam Ramadan is an SEO and GEO Specialist with 10 years of experience. He writes and reviews the guides published by SEO Sharks, an agency working on SEO, link building and AI search visibility.

Robots.txt Optimization: Rules, Examples and Key Mistakes - Robots.txt Optimization: Rules, Examples and
1

A free month of SEO for your website

We offer a free month of SEO that covers everything: technical analysis, competitor analysis, keyword research, your business goals, SEO fundamentals, on-page SEO and AI search optimization (GEO). All the work is done by a dedicated team.

Start my free month
Free consultation via WhatsApp
تم التصميم والسيو بواسطة قروش السيوواتساب +20 112 057 7167أعلن معنا