Shiklo Tools
PDF40
Image20
Calculators & Converters25
Text & Writing13
Developer26
Video14
Audio14
Document14
Data & Spreadsheet12
Web & SEO9
OCR & Scanning5
Archive & File12
Security & Privacy23
Health & Medical29
QR & Barcode5

Robots.txt Generator & Tester

Create a robots.txt with rules per crawler, AI crawler blocking and sitemap lines, then test any URL against any robots.txt file.

A robots.txt file sits at the root of your website and tells crawlers which parts of the site they may visit. The Robots.txt Generator builds one for you: allow everything, block everything, or block specific folders; add rules for individual crawlers such as Googlebot-Image or Bingbot; block the crawlers that collect content for AI training with one switch; and list your sitemaps.

The built-in tester then checks any address against the file, or against a robots.txt you paste, and tells you whether a crawler may fetch it and which line decides. It follows RFC 9309, the robots exclusion standard that Google uses, including wildcards and the longest-match rule. Everything runs in your browser.

How to use Robots.txt Generator & Tester

  1. Choose what all crawlers may do: allow everything, block everything or block some folders.
  2. Optionally block AI training crawlers and add rules for specific crawlers.
  3. Add your sitemap address, then copy or download robots.txt and upload it to your site's root.
  4. Test addresses with the tester, against the generated file or your own.

Features

  • Allow all, block all or block a list of folders and pages.
  • Per-crawler rules for Googlebot, Bingbot, Applebot, DuckDuckBot and more.
  • One switch to block AI training crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot and others.
  • Sitemap lines and an optional Crawl-delay.
  • Tester that follows RFC 9309: most specific user-agent, longest match, Allow wins ties.
  • Supports * and $ wildcards and percent-encoded paths.
  • Checks pasted files for unknown fields and misplaced rules.
  • Download robots.txt ready to upload.

How robots.txt rules work

Rules are grouped under one or more User-agent lines. A crawler uses the group whose user-agent name matches it most specifically, and only that group; if none matches, it uses the group for *. Within the group, the rule with the longest matching path wins, and if an Allow and a Disallow rule are equally long, Allow wins.

RulesAddressResult
Disallow: /admin//admin/settingsBlocked
Disallow: /admin/ and Allow: /admin/public//admin/public/helpAllowed: the Allow rule is longer
Disallow: /*.pdf$/files/report.pdfBlocked: * matches any characters, $ marks the end
Disallow: /*.pdf$/files/report.pdf?download=1Allowed: the address doesn't end in .pdf
Allow: /$ and Disallow: //Allowed: only the home page is open
Disallow: (empty)anythingAllowed: an empty Disallow allows everything

Blocking AI crawlers

Several companies run separate crawlers to collect web pages for training AI models, and most of them respect robots.txt. Turning on the AI switch adds a group that disallows the whole site for GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's Gemini training), Applebot-Extended (Apple), CCBot (Common Crawl), meta-externalagent (Meta), Bytespider (ByteDance) and Amazonbot. Blocking Google-Extended and Applebot-Extended does not affect your ranking in Google Search or Apple's Siri and Spotlight results. Crawlers that fetch pages for AI search answers, such as OAI-SearchBot or PerplexityBot, are separate; you can add rules for them individually.

What robots.txt can't do

robots.txt is a request, not a lock. Well-behaved crawlers follow it, but anyone can still open the pages, and blocked pages can still appear in search results without a description if other sites link to them. To keep a page out of Google, allow it to be crawled and add a noindex meta tag instead, and protect private areas with a password. Never list secret folders in robots.txt, since the file itself is public.

Where to put the file

Upload the file as robots.txt to the root of each host, for example https://example.com/robots.txt; a file in a subfolder is ignored, and www.example.com and shop.example.com each need their own. It must be plain UTF-8 text. Google reads up to 500 KiB of it and caches it for up to 24 hours, so changes take a day to apply. Google ignores Crawl-delay; Bing and Yandex respect it.

Frequently asked questions

How do I block all crawlers from my site?

Choose Block everything. The file then contains User-agent: * and Disallow: /, which asks every crawler to stay away.

How do I block GPTBot and other AI crawlers?

Turn on Block AI training crawlers. It adds a Disallow: / group for GPTBot, ClaudeBot, Google-Extended, CCBot and other known AI training crawlers.

Will blocking AI crawlers hurt my Google ranking?

No. Google-Extended only controls use for Gemini training; Googlebot, which indexes your site for Search, is not affected.

Why is a blocked page still in Google?

robots.txt stops crawling, not indexing. If other pages link to it, Google can list the address without content. Use a noindex tag and allow crawling to remove it.

Does the tester work exactly like Google?

It follows RFC 9309, the standard Google's own parser implements, including user-agent matching, longest match and wildcards.

Where do I put my sitemap?

Add a Sitemap line with the full address of your sitemap, such as https://example.com/sitemap.xml. It can appear anywhere in the file.

Is robots.txt case-sensitive?

Paths are: Disallow: /Admin does not block /admin. Field names and user-agent names are not case-sensitive.

Related tools