BrandPickedBot

BrandPickedBot is the crawler of BrandPicked. It reads a website when someone adds it to their BrandPicked account as their product, or asks us to read it again. It also opens the pages an AI assistant cited when it answered a buyer’s question in a BrandPicked report, to see which products those pages name. And when anyone checks a website with our free AI crawler checker, it reads that website’s robots.txt and nothing else.

What it reads

• The page it was given and up to 11 pages of the same site linked from it, such as pricing, features, integrations and comparison pages.
• For a report, the first screen of the product’s own page, opened in a browser on a laptop and on a phone, and the site’s robots.txt, structured data and llms.txt. The browser runs the page’s scripts, but loads only what robots.txt allows and only through our crawler.
• For a scan, the tags search engines read on the product’s home page, such as its title, meta description and canonical link, and the sitemap its robots.txt names or the one at /sitemap.xml.
• For a report, up to 50 pages that an AI assistant’s answers cited, and nothing linked from them.
• For the AI crawler checker, only the site’s robots.txt. A result is reused for 10 minutes, so checking the same site again within that time reads nothing.
• Only public pages. It never signs in, fills in forms or follows links to other sites.
• A few pages at a time, and when it has several pages of one site to open, it spreads them out.

How to recognise it

It sends this user agent:

BrandPickedBot/1.0 (+https://brandpicked.ai/bot)

How to stop it

It follows robots.txt. To keep it off your whole site, add these lines to your robots.txt. If robots.txt blocks a page, we don’t read it. It still reads robots.txt itself, because that is how any crawler learns your rules.

User-agent: BrandPickedBot
Disallow: /

What we keep

The text of the pages it read, with email addresses and phone numbers removed, for 30 days. We use it to draft a description of the product for the account that asked. We blank email addresses and phone numbers on the first screen before we capture it, and we never store the screenshots. For a page an AI assistant cited, we keep only its address, its title and which of the report’s products it names, never its text. For the AI crawler checker, we keep the site’s address and which AI crawlers its robots.txt lets read the home page, with the rule that decided, for 30 days, and never the robots.txt itself.

If your name is on a page it reads

Public pages can name people, such as a product’s team or an article’s author. We read them only to describe the product and to see which products a page names, on the basis of our legitimate interest, and we keep nothing about the people on them beyond the page text above. You can object by writing to us, or keep the crawler off your site with robots.txt. The privacy notice explains how we handle personal data and your rights.

Questions or a problem with BrandPickedBot? Write to support@brandpicked.ai.