Blocking every AI bot is a publishing decision. So is allowing every one of them. Make that decision deliberately; a training crawler and a bot that retrieves your article for a search answer do different jobs.
You can allow some search crawlers while declining some training uses. You can also accidentally remove your site from an answer engine while believing you’ve only blocked model training. The names matter.
This AI crawler list separates actual HTTP clients from usage-control tokens. It covers 19 documented crawlers and fetchers, links to official IP sources, and gives you robots.txt examples you can adapt to your own site.
Crawler permissions are one part of a wider LLMO strategy. This guide focuses on access; use the strategy guide for the content and measurement work around it.
Last documentation review: . The verification dates below refer to documentation checks, not independent tests of crawler behavior.
Which AI bots should you allow?
If you want visibility in AI search, review access for OAI-SearchBot, Claude-SearchBot, PerplexityBot, MistralAI-Index, Amzn-SearchBot and DuckAssistBot. Decide separately whether to permit training-oriented collection.
Don’t assume that blocking training prevents search retrieval. Don’t assume that allowing search grants permission for every other use. Each operator defines its own controls; the table records those distinctions.
One limit deserves attention before you edit anything: RFC 9309 defines robots.txt as a protocol for cooperative crawlers, not an access-security system. Protect private content with authentication and authorization.
AI crawler list: Operators, user agents and official IP sources
The HTTP column shows the identifying user-agent marker, not a complete browser-style header. Versions and surrounding browser strings can change. The robots.txt column identifies the policy token; it doesn’t promise that a user-triggered fetcher will honor a denial.
Purpose key: Search means discovery or retrieval for search answers. Training means collection that may support model training. User action means a request initiated through a product user. Dataset collection means an archive that downstream organizations may reuse.
Documented crawlers and fetchers relevant to AI visibility, reviewed November 3, 2026
Googlebot and bingbot belong here because traditional search infrastructure matters to AI discovery. Calling either one a dedicated training bot would misrepresent its role. CCBot belongs in the dataset category: Common Crawl publishes crawl data; it’s not a consumer answer engine.
Amazon’s linked IP sources are web pages containing address data, rather than ordinary standalone JSON endpoints. Don’t assume every source in this table accepts the same parser.
Google-Extended and Applebot-Extended aren’t crawlers
A control token can appear after User-agent: in robots.txt without ever appearing in an HTTP request. That’s the distinction that makes many AI bot lists misleading.
Usage controls, kept separate from HTTP crawler identities
Operator and token
What it controls
Separate HTTP user agent and IP file?
Official source
Last checked
Google: Google-Extended
Future Gemini model training and specified grounding uses in Gemini Apps and Vertex AI. It doesn’t control inclusion or ranking in Google Search.
No separate HTTP user agent; no separate crawler IP feed.
Training-use preference for adopting operators. Microsoft has announced support from February 1, 2027; Google states an intention to support it in early 2027.
A usage signal, not an HTTP client. Don’t treat it as a universal active control in November 2026.
If your log tool reports a visit from Google-Extended, inspect what it actually recorded. A policy label assigned by the tool isn’t proof of an HTTP request carrying that identity.
Blocking Google-Extended also affects specified grounding uses
Don’t describe Google-Extended as a training-only switch. Its documented scope includes grounding in certain Google products. Google Search AI Overviews and AI Mode use Search controls, which Google explains in its AI features guidance.
HTTP user agents versus robots.txt tokens
An HTTP User-Agent header identifies the client making a request. A robots.txt product token selects a policy group. For many bots, the names match. The complete strings don’t.
For example, Mistral documents this full HTTP header value for its user-triggered fetcher:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)
The corresponding robots.txt group uses the product token:
You don’t paste the browser wrapper into the User-agent: line. You also shouldn’t hardcode a browser version as the only way your firewall recognizes a bot. Keep identity verification separate from version matching.
There isn’t one ChatGPT crawler
OpenAI separates search access through OAI-SearchBot from potential training collection through GPTBot. ChatGPT-User handles certain user actions; it isn’t the search opt-out control. OpenAI says robots.txt may not apply to those user-initiated requests.
That last point matters. A Disallow entry isn’t a dependable network block for every client carrying an AI product’s name. Check the operator’s documented behavior before treating it as one.
User-triggered doesn’t mean the same robots.txt policy everywhere
Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says its bots, including Claude-User, honor robots.txt. Amazon says Amzn-User may not follow every directive. Apply the documented policy for the actual client rather than extending one operator’s exception to everyone else.
How to set robots.txt rules for AI bots
Choose the business policy first. Do you want your public research available for search answers? Are you comfortable with training collection? Is a paid archive subject to a different policy?
Then write the rules. A long blocklist won’t answer those questions for you.
Allow selected AI search crawlers; decline selected training and dataset collection
This example creates a shared policy for six search agents and a separate policy for five training or dataset collectors. It doesn’t set a policy for every operator, every user-triggered fetcher, or your existing Googlebot and bingbot groups.
Merge it into your current file. If a named group already exists, edit that group and check for conflicting rules. Replace the illustrative path exclusions with the public-site exclusions you actually need.
If you also want user-requested retrieval, review the user-action rows separately. Allowing Claude-SearchBot doesn’t automatically allow Claude-User when another applicable rule denies it.
Decline the documented Google and Apple usage controls
These groups express usage preferences rather than denying a separate HTTP crawler:
Read the scope table before adding them. In particular, a Google-Extended denial also declines the specified grounding uses; don’t add it to a policy described internally as “training only.”
Block a selected search crawler
If you decide against a particular search integration, give that crawler an explicit denial. For example:
User-agent: PerplexityBot
Disallow: /
That addresses PerplexityBot’s automated collection. It doesn’t turn the separate Perplexity-User identity into a reliably robots-compliant client. Enforce a required access denial at the server or edge, using verified identities where possible.
Keep path restrictions inside the applicable group
Googlebot’s specific group doesn’t inherit the wildcard group’s /account/ restriction. If that exclusion should apply, put it in the Googlebot group too. Google’s robots.txt specification guide explains group selection and rule precedence.
Within a group, a more specific path rule can override a broader one. Don’t treat the file like a firewall where the first matching line always wins.
Make fallback behavior explicit
Apple says Applebot follows Googlebot directives when no Applebot group exists. Amazon documents fallback behavior for Amzn-SearchBot when its own token is absent and other search bots are allowed. If you need a different policy, name the relevant agent explicitly instead of relying on the wildcard alone.
Bing’s October 2026 announcement: Prepare for 2027
On October 30, 2026, Microsoft announced separate training and grounding preferences. They begin taking effect on February 1, 2027 as Microsoft crawls. They’re announced controls at this article’s review date, not a feature already active in November 2026.
The announced training signal includes:
User-agent: __usage_ai-training
Disallow: /
Microsoft also documents page-level training and grounding controls. One announced grounding example is:
This is page metadata, not a robots.txt rule. Microsoft says legacy NoArchive and NoCache AI preferences stop applying after March 1, 2027. Existing users need to follow its transition instructions, including the temporary Bing Webmaster Tools grounding preference where applicable.
How to verify an AI crawler and check whether it can access your site
A user-agent string is a claim. Anyone can send one.
Your verification process needs the actual request source and the operator’s current evidence. Otherwise, an allow rule for a trusted bot can become an allow rule for anyone who copies its name.
Compare the source IP with the correct official list
Start with the published IP source in the reference table. Match CIDR ranges properly, including IPv6 when the feed provides it. A string-prefix comparison isn’t a subnet check.
Use the list for the claimed bot, not just another bot from the same company. Perplexity publishes separate feeds for PerplexityBot and Perplexity-User. Anthropic’s shared feed verifies an Anthropic source; it doesn’t, by itself, distinguish ClaudeBot from Claude-SearchBot.
Don’t substitute an entire cloud provider’s address space for a bot-specific list. Hosting on Amazon Web Services or Azure doesn’t prove that a request belongs to an approved crawler.
For CCBot, Common Crawl also documents forward-confirmed reverse DNS for IPv4. Follow the operator’s exact method; a hostname that merely contains a familiar brand name proves nothing.
Check the request at the CDN and at the origin
Record the requested host and path, timestamp, user-agent header, trusted client IP, response status and security action. If a CDN rejects a request before it reaches WordPress, your origin access log won’t tell the whole story.
Use client-IP data supplied through a trusted proxy configuration. Don’t build verification around an arbitrary forwarding header that an external visitor can set.
When an approved bot gets blocked, adjust the specific challenge or bot rule responsible. Keep any exception scoped to the public content and request methods it needs; authentication still applies.
Test the delivered page, not just the robots.txt text
Start with a normal GET request:
curl -sS -D - https://example.com/robots.txt
Then inspect a public article. The following uses a synthetic user-agent marker for diagnosis; it does not impersonate a verified source IP:
This can expose a rule that reacts to that header. It can’t prove that genuine crawler traffic will receive the same response. Confirm that in verified request logs.
A 200 status isn’t enough either. Open the response body. A challenge page can return 200 while the article never arrives. Check that the main answer and reference table appear in the delivered content, and inspect any X-Robots-Tag headers or page-level robots directives.
Keep the verification data current
Refresh IP lists on a defined schedule; validate the response before changing production rules. Keep the last valid version when a fetch fails and investigate the failure. Replacing a working list with an empty response creates a new problem.
Recheck the operator documentation when a new identity appears. An unfamiliar bot name shouldn’t silently inherit your approval for every other product that company operates.
What crawler access means for SEO and LLM visibility
Access gives a system a chance to retrieve your work. It doesn’t give your work a reason to be cited.
For Google AI Overviews and AI Mode, Google says a supporting page must be indexed and eligible for a Search snippet. It doesn’t require a special AI schema or a new AI text file. Its published guidance points publishers back to established SEO requirements.
If you’re weighing up llms.txt, our llms.txt SEO and GEO analysis explains the proposal and the limits of our implementation observations. Keep that discussion separate from the crawl permissions you set in robots.txt.
That leaves plenty of useful work. Put the actual answer in readable text. Link to original evidence. Give tables meaningful headers. Explain what you checked, and distinguish a documented policy from something you observed yourself.
For a reference page like this one, dates are part of the content. A reader needs to know whether “supports this control” means active today or announced for next February. An answer system needs that distinction too.
Keep access, citations and business results in separate reports
Use verified server or CDN logs to measure crawler access. Check real answers to measure whether your page gets cited. Use analytics and conversion data to measure what those visits produce.
Our SEO and GEO case study presents organic traffic and captured AI citations as separate observations, rather than proof that one caused the other.
A crawler request isn’t a referral visit. A referral visit isn’t proof that model training produced the recommendation. Keep those claims separate and your reporting becomes much more useful.
Record the exact question when you check an AI answer, together with the date, product, language and cited URL. Repeat the same checks after a meaningful change. One screenshot can document one answer; it can’t establish a stable visibility trend.
For a worked example, see our B2B AI visibility benchmark. It records a fixed set of questions across multiple platforms and explains the limitations of a single test run.
Frequently asked questions about AI bots and robots.txt
Can I block GPTBot and still appear in ChatGPT search?
Yes. OpenAI documents independent controls for GPTBot and OAI-SearchBot. Allow the search identity if you want search eligibility, and make the training decision separately. Eligibility doesn’t guarantee a citation.
Does blocking Google-Extended remove my site from Google AI Overviews?
No. Google-Extended doesn’t control Google Search inclusion. Google documents Googlebot access and Search preview controls for its Search AI features; the Extended token has a different usage scope.
Does allowing a training crawler improve AI search visibility?
Don’t treat training permission as a citation tactic. Search retrieval and model training are different processes. Where an operator offers separate controls, evaluate each permission against its documented purpose.
Does noindex block AI training?
Don’t use it as a universal training opt-out. An indexing directive and a training-use preference address different questions. Use the relevant operator’s documented controls for the use you want to restrict.
Can robots.txt stop every AI scraper?
No. Cooperative clients can honor it; other clients can ignore it. Where access must be denied, enforce that decision at the application, server or edge.
How long do robots.txt changes take to affect AI bots?
There’s no universal processing time. DuckDuckGo documents 72 hours for a DuckAssistBot opt-out. Don’t apply that timing to a different operator, and don’t interpret a cached policy as proof that a new rule failed.
Will blocking a crawler erase information a model already learned?
A new robots.txt rule isn’t a model-deletion request. Treat it as a forward-looking preference for the uses the operator documents, and use any separate removal process when you need to address existing content.
Sources, scope and verification method
This reference uses operator documentation and operator-published IP sources. The table distinguishes HTTP identities from usage tokens, and labels future controls separately. “No IP feed linked” means the reviewed page didn’t provide one; it doesn’t prove that no verification mechanism exists anywhere.
Coverage is selective. Meta’s crawler documentation couldn’t be independently retrieved during this review, so its agents aren’t included in the verified table. No bot-specific IP feed or current behavior has been inferred from a third-party directory.
The rules are policy examples, not results from a production crawl experiment. Before deploying them, check the combined file against your actual hosts, paths and existing groups. Keep robots.txt available at the root of each relevant host.
Make the choice explicit. Keep the evidence current. Then give people something useful enough that a search engine, an AI assistant or another human has a reason to send the next reader your way.
Need help finding access issues on your own site? Our GEO audit and AI visibility monitoring covers technical accessibility alongside the sources and answers that matter to your business.
With years of experience navigating the ever-evolving crypto landscape, Eugen knows exactly how to make content shine in Google’s eyes—without breaking the algorithm. With experience working as an SEO specialist in real fast-growing crypto companies, along with training in crypto trading, Google Ads Search Certification, and Google Analytics Individual Qualification, he is a master of SEO in the crypto world, blending AI-powered strategies with deep industry knowledge. From ChatGPT to blockchain trends, he knows how to make content rank, engage, and convert.
