Skip to content
BrandPicked

AI crawler checker

Enter your website to see whether its robots.txt lets AI crawlers read your home page, and which rule decides for each one. It's free and needs no account.

We read only its robots.txt, as BrandPickedBot, and nothing else.

What it checks

AI crawlers do three different jobs, and a robots.txt can treat each one differently. We check every crawler below against your home page.

  • Answering a person's question
    These fetch your pages when someone asks an AI assistant a question and it opens your site to answer.Claude-UserChatGPT-UserPerplexity-User
  • Building AI search indexes
    These index your pages so AI assistants and AI search can find and cite them.Claude-SearchBotOAI-SearchBotPerplexityBot
  • Collecting training data
    These collect pages to train AI models. Blocking them doesn't stop assistants from reading your pages when they answer.ClaudeBotGPTBotGoogle-Extended

What it can't tell you

  • robots.txt is a request. Whether a crawler follows it is up to that crawler.
  • A firewall or bot protection can still turn crawlers away even when robots.txt lets them in. We only read robots.txt.
  • We check the address you enter. If your site redirects to another host, such as www, check that one too.

We keep each result for 30 days at a link that anyone who has it can open. The privacy notice explains what we keep. Read the privacy notice