How to rank in Claude

To rank in Claude, a page has to be found by Claude's web search and then chosen as a source. Anthropic lists Brave Search and TurboPuffer as its web search subprocessors (opens in a new tab), and it runs its own crawlers: Claude-SearchBot, which indexes pages for search (opens in a new tab), and Claude-User, which fetches pages when someone asks Claude a question (opens in a new tab). Allow both crawlers, make sure Brave can crawl the page, and keep key facts in server-rendered text.

Anthropic documents which crawlers and search providers are involved. It does not document how Claude chooses which results to cite, so everything after crawler access comes from outside studies. This guide keeps the two apart. It is one of a set on how AI assistants find and cite sources, and its sources were checked on September 27, 2026.

Scope and definitions

Claude is Anthropic's AI assistant. It answers some questions from what it learned in training and others by searching the web. Work to be cited in AI answers is called answer engine optimization (AEO) or generative engine optimization (GEO). This guide covers both routes into Claude's answers, with most attention on web search, because a new page can appear there without waiting for a new model.

• Parametric knowledge is what a model learned in training. Anthropic gives reliable knowledge cutoffs of June 2026 for Claude Opus 5.5 and Claude Fable 5.1, January 2026 for Claude Sonnet 5, and February 2025 for Claude Haiku 4.5 (opens in a new tab). A company or page that appeared after a model's cutoff can reach that model's answers only through search.
• Web search is the tool Claude uses to look things up while answering. Anthropic's API documentation says Claude searches when a request depends on information that is "current, changing, or outside its training data," including "information about specific organizations, people, or products that might have changed," and answers directly when a request draws on stable knowledge (opens in a new tab).
• A subprocessor is a company that processes data on Anthropic's behalf. Anthropic publishes its list in its Trust Center.
• Retrieval is finding and fetching documents to answer a question. A page that is not retrieved cannot be cited as a source.
• robots.txt is the file at the root of a site that tells crawlers which pages they may fetch. Its rules are standardized in RFC 9309 (opens in a new tab).

How Claude finds and cites pages

Where Claude's search results come from

Anthropic's Trust Center lists two subprocessors whose purpose is web search (opens in a new tab) (official documentation):

No other subprocessor on the list has web search as its purpose; Google Cloud Platform and Microsoft Azure appear, but for cloud infrastructure (opens in a new tab). The API documentation for the web search tool does not name a search provider (opens in a new tab).

A secondary source adds history. In March 2025, a developer reported that Anthropic had added Brave Search as a subprocessor on March 19, 2025, that Claude's search tool exposed a parameter named "BraveSearchParams," and that Claude's results for a test query matched Brave's (opens in a new tab) (secondary source from 2025; the Trust Center listing above is the current confirmation).

What the documentation does not say:

• how Brave Search, TurboPuffer, and Anthropic's own crawler fit together;
• how much of Claude's search comes from each;
• what TurboPuffer does beyond the listed purpose, "Web Search."

Our reading, which Anthropic has not confirmed, is that Claude may combine Brave's index with an index Anthropic builds itself using Claude-SearchBot. Treat that as a hypothesis.

Anthropic's three crawlers

Anthropic's help center describes three agents. All three honor robots.txt, and Anthropic also supports the non-standard Crawl-delay rule (opens in a new tab) (official documentation, updated April 7, 2026):

Anthropic publishes its crawlers' IP addresses at claude.com/crawling/bots.json (opens in a new tab), so a firewall can confirm that a request really comes from Anthropic.

Two points follow from the table. First, ClaudeBot concerns training, not search: blocking it does not, by Anthropic's description, remove a site from Claude's search results. Second, Claude-User is different from Perplexity's user agent, which Perplexity says generally ignores robots.txt (opens in a new tab); Anthropic says a robots.txt block on Claude-User stops it retrieving your content. The Perplexity side is covered in how to rank in Perplexity.

Brave's crawler

Brave Search is the first of the two web search subprocessors. Three facts about its crawler matter here (official documentation):

• It does not advertise a separate user agent (opens in a new tab), so you cannot allow or block it by name in robots.txt.
• If a domain or page is not crawlable by Googlebot, Brave's crawler will not crawl it either (opens in a new tab). Rules you write for Googlebot therefore also apply to Brave.
• Its discovery is partly powered by data from Brave browser users who opt in to the Web Discovery Project (opens in a new tab).

A page you block for Googlebot is therefore invisible to Brave, and so to the part of Claude's search that uses Brave.

The Brave URL submission form

Brave has a form at search.brave.com/submit-url. Brave documents it for two uses: re-fetching a page after you add a noindex rule so the page can be delisted, and delisting a page that no longer exists (opens in a new tab). Brave does not say that submitting a new URL adds it to the index or speeds its discovery, and we found no evidence either way, so this guide does not recommend using the form for new pages.

When Claude searches, and what it receives

Questions about a specific company's current prices or offers are the kind the documentation says trigger a search: "current prices, rates, scores, or statistics" and information about specific organizations or products that might have changed (opens in a new tab). Each search result carries the page's URL, its title, and a page_age field that the documentation describes as when the site was last updated, and citations are always enabled (opens in a new tab). This is the API documentation; Anthropic does not separately document how search works inside the Claude apps.

What research shows about Claude's citations

Few studies test Claude directly. These findings include it:

• Earned media leads. In API tests with ranking-style prompts, 86% to 87% of the domains Claude cited were earned media, such as third-party reviews and editorial coverage, and Claude's set of cited domains was the most stable across languages (opens in a new tab) (preprint; Claude 3.5 Sonnet with web search; mid-2025).
• Rewriting tricks did not help. A 2025 benchmark found statistically significant gains for conversational-SEO methods in 3 of 54 cases, and adding statistics lowered rank in 19 of 24 (opens in a new tab) (peer-reviewed). On Claude 3.5 Haiku, no method improved rank, and many lowered it significantly (opens in a new tab).
• Concrete facts matter in the lab. In 252,000 controlled trials across six models, including Claude 3.5 Sonnet, an explicit price and a recent date raised the odds of being cited first in all six, while formatting alone had no consistent effect (opens in a new tab) (peer-reviewed; laboratory; authors employed by a software vendor).
• Manipulation failed in one test. An endorsement-manipulation attack succeeded 0.0% of the time against Claude Sonnet 4.6, against up to 31.4% for Gemini 3 Flash (opens in a new tab) (preprint; we read the abstract only).
• Answers vary between runs. When the same prompt was repeated, ChatGPT and Google's AI returned the same list of brands less than once in 100 runs, and Claude only slightly more often (opens in a new tab) (2,961 runs, November–December 2025; vendor study with a noted conflict of interest).
• Small brands start behind. Across engines including Claude, on a first answer to a generic prompt, household brands appeared 73% of the time, mid-market brands 44%, and small brands 11% (opens in a new tab) (preprint; the author co-founded the tracking platform studied).

How much traffic Claude sends

In one mostly B2B referral panel, Claude's share of AI referrals rose from 1.4% to 18.5%, and it rose at 14 of the 16 brands with enough volume (opens in a new tab), between mid-2025 and March to April 2026 (vendor panel). Anthropic also crawls far more than it refers: in July 2025, it made 38,065 page requests per referral, down from 286,930 in January 2025 (opens in a new tab) (network measurement). Those ratios may overstate the gap, because traffic from Claude's native app does not carry a referrer header (opens in a new tab), so some visits from Claude are never counted as referrals.

Steps

Steps 1 to 4 follow Anthropic's and Brave's documentation. Steps 5 to 9 are recommendations based on the evidence above.

1. Decide on the three crawlers separately. To appear in Claude's search answers, allow Claude-SearchBot and Claude-User. ClaudeBot is a separate decision about training. Blocking it signals that your future content should be excluded from training (opens in a new tab); whether allowing it makes a brand more likely to be recommended has not been measured.
2. Check which robots.txt group each crawler follows. A crawler obeys the group that names it and uses the * group only when no group matches (opens in a new tab). If your * group already allows your public pages, you need nothing more. If you add named groups, such as the example below, repeat in them any Disallow lines you still need, because the named groups replace your * rules for those crawlers.

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

3. Keep Googlebot allowed on pages you want Claude to find. Brave's crawler skips anything Googlebot cannot crawl (opens in a new tab).
4. Check your host and CDN. Some hosts block crawlers with one setting. On Vercel, for example, the AI Bots and Bot Protection rulesets are off by default but can deny or challenge bot traffic when switched on (opens in a new tab). Match requests against Anthropic's IP list, and confirm in your server logs that Claude-SearchBot and Claude-User receive HTTP 200.
5. Check Brave coverage. Search Brave for your brand and your key pages. If they are missing, confirm that Googlebot can crawl them.
6. Put key facts in server-rendered HTML. In Vercel's network data from December 2024, none of the major AI crawlers rendered JavaScript; Claude's crawler fetched JavaScript files in 23.84% of requests but did not run them (opens in a new tab). In a practitioner test, no AI system read facts that existed only in JSON-LD when it fetched a live page (opens in a new tab) (one test page).
7. Publish the facts that search-triggering questions ask for. State current prices, what is included, and who the offer is for on crawlable pages, and keep a true "last updated" date on them.
8. Earn third-party coverage. Claude's citations lean heavily on earned media. Seek independent reviews and editorial coverage. For how mentions relate to AI answers, see brand mentions and AI recommendations.
9. Measure with several instruments. Ask Claude, with web search on, a fixed set of buyer questions several times over several weeks. In analytics, note that Google's May 2026 release note for GA4's AI Assistant channel names Claude (opens in a new tab), but the channel definition lists ChatGPT, Gemini, Deepseek, Copilot, and Grok (opens in a new tab), so a custom channel for claude.ai is safer; Microsoft Clarity's AI channel names Claude (opens in a new tab). Count Claude-SearchBot and Claude-User requests in your server logs. Methods are covered in how to measure AI visibility and how to track AI referral traffic.

These steps do not cover other assistants. ChatGPT search, for example, depends on OpenAI's own crawler and on search partners that include Microsoft (opens in a new tab).

What the evidence shows and does not show

Antipatterns

An antipattern is a practice that looks helpful but fails or backfires.

1. Confusing ClaudeBot with Claude's search. ClaudeBot concerns training (opens in a new tab), so blocking it does not, by Anthropic's description, remove you from search results, and allowing it is not a substitute for allowing Claude-SearchBot and Claude-User. Instead, decide on each crawler separately.
2. Blocking Claude-SearchBot or Claude-User and expecting citations. Anthropic says blocking them may reduce your visibility in user search results (opens in a new tab). Instead, allow both.
3. Blocking Googlebot on sections you want in Claude. Brave will not crawl what Googlebot cannot (opens in a new tab). Instead, keep one consistent set of rules for public pages.
4. Adding named groups that silently drop your * rules. A crawler that matches a named group ignores the * group (opens in a new tab). Instead, repeat the Disallow lines you need.
5. Treating the Brave form as a way in. It is documented for re-fetching and delisting (opens in a new tab). Instead, make pages crawlable and linked.
6. Keeping facts in JSON-LD or scripts. Live fetchers ignored JSON-LD-only facts (opens in a new tab) in one test. Instead, put facts in visible HTML.
7. Rewriting pages with GEO heuristics. Rewrites often lowered rank on Claude 3.5 Haiku (opens in a new tab). Instead, add specific, true facts.
8. Hidden instructions to AI. They failed against Claude Sonnet 4.6 in one test (opens in a new tab), and Google's spam policies cover attempts to manipulate generative AI responses (opens in a new tab). Instead, state facts plainly.
9. Judging by referrals alone. The Claude app sends no referrer header (opens in a new tab). Instead, combine referrals with logs and repeated tests.

Checklist

FAQ

Does Claude use Google or Bing to search the web?

Anthropic's subprocessor list names Brave Search and TurboPuffer for web search (opens in a new tab). Google Cloud Platform and Microsoft Azure appear on the same list for cloud infrastructure, not for search (opens in a new tab).

If I block ClaudeBot, will Claude stop citing my site?

Not by Anthropic's description. ClaudeBot collects content for training (opens in a new tab). Anthropic ties search visibility to Claude-SearchBot and Claude-User, and Claude's search also draws on Brave Search (opens in a new tab).

Will submitting my URL to Brave get my page into Claude?

Unknown. Brave documents its form for re-fetching and delisting (opens in a new tab), not for adding new pages. IndexNow, a protocol for notifying search engines of new URLs, lists Bing, Yandex, Naver, Seznam.cz, Yep, and Amazon as participants (opens in a new tab); Brave is not listed.

How can I tell whether Claude has read my pages?

Check your server logs for the Claude-SearchBot and Claude-User user agents, and match the requesting IP addresses against Anthropic's published list (opens in a new tab).

Does Claude read llms.txt?

There is no sign that its search does. Claude-SearchBot made no significant number of llms.txt requests (opens in a new tab) in one vendor's server logs.

Why doesn't Claude mention my company?

Three documented reasons are possible. The company may be newer than the model's knowledge cutoff (opens in a new tab). Claude may have answered without searching (opens in a new tab). Or better-known brands took the places: across engines including Claude, small brands appeared in 11% of first answers to generic prompts (opens in a new tab).

How long until Claude cites a new page?

No study has measured it.

Change log

• September 27, 2026: First published.

Next steps

To check whether AI crawlers can reach your pages and how AI assistants answer your buyers' questions, AEO HQ's Instant AEO Audit ($499) is an automated audit of your site and up to three competitors. It includes a live crawl of your key pages and checks of AI crawler access, schema, and llms.txt.