Robots.txt Checker

Check your robots.txt

Check if you are preventing any AI crawlers or Search Bots from seeing your pages.

Input

Choose a mode: fetch robots.txt, paste it, or quickly check a page’s status.

What is robots.txt?

Robots.txt is a plain-text file at the root of your site (e.g. https://yoursite.com/robots.txt) that tells crawlers which paths they are allowed or not allowed to request. It controls crawling, not indexing: it guides where bots spend their crawl budget. It does not hide content from search results by itself.

What is robots.txt and how it controls search engine crawling for SEO — our full guide to syntax, use cases, and common mistakes.

How to use this tool to detect robots.txt blocks and allows

  1. Choose a mode: Fetch from site (we load /robots.txt from a URL you enter) or Paste robots.txt (you paste the file content). Use Check URL status to only test whether one page is allowed or blocked for a user-agent.
  2. Enter the site (e.g. example.com) or paste the file, then click Run & Check.
  3. Review the report: robots.txt URL analysed, AI crawlers blocked, other user agents blocked, and the Disallow report. Use the optional Test URL and User-agent in Advanced to see the exact rule that applies to a specific page.

What are the reports you see and what they mean

Report (summary): Shows the robots.txt URL we analysed, plus counts of groups, rules, sitemaps, and warnings. Pills show how many AI crawlers are allowed vs blocked for the test URL.

AI crawlers blocked / Other user agents blocked: Lists crawlers that are disallowed for the test URL and the matched rule (line and pattern). User-agent is the crawler identifier (e.g. GPTBot, Googlebot). Disallow means the crawler is asked not to fetch that path; Allow can override a broader Disallow for a specific path.

URL allow/block: When you supply a test URL and user-agent, this shows whether that URL is allowed or blocked and which rule (Allow/Disallow and pattern) matched.

Disallow report: Every User-agent group in your robots.txt and its Disallow and Allow rules. Sitemap lines tell crawlers where your XML sitemap is. Expand a row to see the rules for that group.

Understanding AI and LLM agents visiting your website

Many AI products (ChatGPT, Claude, Perplexity, etc.) use crawlers to collect web content for training or retrieval. These crawlers often send a distinct user-agent and many honour robots.txt. If your robots.txt disallows a path for that user-agent, they typically will not request that URL. Knowing which AI agents can or cannot crawl your site helps you align crawling with your policy (e.g. allowing discovery vs limiting use of your content).

How to check if AI crawlers are blocked or not

Use this tool: fetch or paste your robots.txt, then look at AI crawlers blocked and URL allow/block. This tool checks whether the following agents are allowed or blocked for the URL you test:

  • OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
  • Anthropic: ClaudeBot, Claude-Web
  • Google: Google-Extended
  • Perplexity: PerplexityBot
  • Common Crawl: CCBot
  • TikTok: ByteSpider (Bytespider)

Crawler names and user-agents change over time; we use a best-effort list. For a specific page, use Check URL status or the Advanced Test URL and pick the user-agent to see the exact result.

Should you block AI crawlers or not?

It depends on your goals. Blocking them (via Disallow in robots.txt) can reduce how much of your content is used for training or AI answers, and may align with privacy or licensing. Allowing them can help your content appear in AI-powered search or assistants. There is no single right answer—use this tool to see your current setup, then decide in line with your content and business policy. Remember: robots.txt only guides crawling; it does not remove pages from search indexes or fully control how AI systems use content.

Frequently Asked Questions