How ChatGPT, Gemini, Claude, Perplexity, and Copilot find and cite sources

When an AI assistant cites web pages, it has usually found them in a search index while writing its answer. Google's AI Overviews and AI Mode use Google's index (opens in a new tab). Copilot uses Bing (opens in a new tab). ChatGPT uses OpenAI's own crawler (opens in a new tab) plus search partners that include Microsoft (opens in a new tab). For Claude, Anthropic lists Brave Search as a web-search subprocessor (opens in a new tab) and runs its own search crawler (opens in a new tab). Perplexity runs its own crawler (opens in a new tab). A page missing from the relevant index cannot be cited through search.

This page is a reference. For each assistant, a table lists the index or search provider behind its answers, its crawlers and what each one is for, the controls that decide whether a page can be included, and the reports each platform gives site owners. The platform facts come from each company's own documentation, checked on September 27, 2026. Research on how the assistants differ when they cite follows the tables. How to act on these facts is covered in AI search optimization.

Guides in this hub

Each assistant has its own guide:

• How to rank in ChatGPT search
• How to get recommended by ChatGPT
• How to rank in Google AI Overviews and AI Mode
• How to rank in Gemini
• How to rank in Microsoft Copilot
• How to rank in Claude
• How to rank in Perplexity

Scope and definitions

This page covers the organic route into AI answers: how assistants find, fetch, and cite public web pages. It does not cover ads or shopping feeds. The work of improving that route goes by several names, including answer engine optimization (AEO) and generative engine optimization (GEO).

• Retrieval. The step in which an assistant runs searches and pulls pages into its working context before it writes. Answers based on those pages are said to be grounded; grounding is the general term.
• Parametric knowledge. What a model learned in training. An assistant that does not search answers from this alone.
• Search index. The store of crawled pages that a search system retrieves from.
• Crawler. A program that fetches web pages. Each crawler announces a name, its user agent, which site owners use in robots.txt rules.
• Training crawler, search crawler, user-triggered fetcher. The three purposes the documentation distinguishes. A training crawler collects content that may be used to train models. A search crawler builds the index used for answers. A user-triggered fetcher opens a page because a user's request needs it.
• robots.txt. A file at the root of a site that tells crawlers what they may fetch. The rules are standardized in RFC 9309 (opens in a new tab).
• Query fan-out. Turning one question into several searches. Google describes it as "issuing multiple related searches across subtopics and data sources (opens in a new tab)."
• First-party reporting. Data that the platform itself gives site owners about their pages.

The five assistants at a glance

"None documented" means no such report appears in the company's documentation as of September 27, 2026.

Two routes into an answer

Assistants answer from training data or from retrieved pages, and they do not search for every question. In a US clickstream panel, ChatGPT ran a web search on 34.5% of queries in February 2026, down from 46% in late 2024 (opens in a new tab) (vendor study). Claude's documentation says it searches "when the request depends on information that is current, changing, or outside its training data" (opens in a new tab), including information about specific organizations, people, or products.

Training data has a cutoff: Anthropic lists June 2026 as the reliable-knowledge cutoff of its newest Claude models (opens in a new tab). It follows that a company that is new, or that recently changed its prices or services, reaches these assistants mainly through retrieval. The tables below describe that route.

ChatGPT (OpenAI)

What research adds. ChatGPT's domain concentration differed significantly from Google's but not from Bing's, which the authors read as reliance on Bing (opens in a new tab) (preprint; July–August 2025). ChatGPT drew 93.5–95.1% of the domains it cited from earned media, such as reviews and editorial coverage (opens in a new tab) (preprint; API versions, mid-2025). Pages ChatGPT cited averaged 958 days old, the newest of the assistants measured (opens in a new tab) (vendor study).

Guides: ranking in ChatGPT's search results and getting recommended by ChatGPT.

Google: AI Overviews, AI Mode, and Gemini

What research adds. In a December 2025 benchmark of more than 11,000 queries, AI Overviews and organic results shared 18% of their sources, and AI Overviews and Gemini shared 11% (opens in a new tab) (peer-reviewed). The same study found that 21 publishers blocking Google-Extended were never cited by Gemini and were less likely to be retrieved by AI Overviews (opens in a new tab), although Google says the token does not affect Search. The link is an association, not a tested cause. 38% of Gemini responses cited no website at all (opens in a new tab) (preprint; July–August 2025). AI Overviews showed no preference for newer pages than organic results (opens in a new tab) (vendor study).

Guides: ranking in Google's AI answers and ranking in Gemini.

Microsoft Copilot

Indexing notes. IndexNow shares each submission with Bing, Yandex, Naver, Seznam.cz, Yep, and Amazon, but not Google, and "Submitting a URL does not guarantee immediate indexing." (opens in a new tab) In 2023, Bing said 12% of new URLs clicked in its results were first discovered through IndexNow (opens in a new tab) (platform figure). Because Copilot searches Bing, a page Bing has not indexed cannot be cited in Copilot's web answers.

Guide: ranking in Microsoft Copilot.

Claude (Anthropic)

What research adds. Claude drew 86–87% of the domains it cited from earned media, and its set of cited domains was the most stable across languages (opens in a new tab) (preprint; API version, mid-2025). How much of Claude's web search comes from Brave's index and how much from Anthropic's own crawl is not documented.

Guide: ranking in Claude.

Perplexity

What research adds. Perplexity's Sonar model consulted about 14 pages per query, against about 9 for AI Overviews and Gemini and about 4 for OpenAI's GPT-4o search model (opens in a new tab) (preprint; 4,706 queries, 2025). Perplexity drew 17.5–23.8% of the domains it cited from social platforms, the most of the engines tested (opens in a new tab) (preprint). Among 112 recently launched startups, those with more referring domains and more Reddit presence were more likely to be found by Perplexity (opens in a new tab) (preprint, thesis-based). Perplexity's documentation does not mention Bing or any other third-party index.

Guide: ranking in Perplexity.

How the assistants differ when they cite

These findings come from independent studies, mostly run through each product's API. They describe tendencies, not fixed rules.

Answers also vary from run to run within one assistant. When the same prompt was repeated, ChatGPT and Google's AI returned the same list of brands less than once in 100 runs, and Claude only slightly more often (2,961 runs, November–December 2025) (opens in a new tab) (vendor study; a co-investigator works for a visibility-tracking vendor), and about 65% of cited sources changed from one day to the next (opens in a new tab) (preprint). Measuring AI visibility therefore takes repeated runs on each assistant.

Rules that apply to every assistant

• The most specific robots.txt group wins. A crawler follows the group that names its user agent, and the * group applies only when no named group matches (opens in a new tab). A crawler with its own group therefore ignores rules added only under *.
• User-triggered fetchers may not follow robots.txt. OpenAI says robots.txt rules "may not apply" to ChatGPT-User (opens in a new tab), Perplexity says Perplexity-User "generally ignores" them (opens in a new tab), and Google says its user-triggered fetchers "generally ignore robots.txt rules" (opens in a new tab). Anthropic says its bots honor robots.txt (opens in a new tab).
• Search and training are separate choices. OpenAI says its crawler settings are independent (opens in a new tab), Google says Google-Extended does not affect inclusion in Google Search (opens in a new tab), and Anthropic describes ClaudeBot (training) and Claude-SearchBot (search) as separate bots with separate effects (opens in a new tab). Google-Extended does govern grounding in the Gemini app.
• Crawlers read HTML, not scripts. In data Vercel published in December 2024, none of the major AI crawlers rendered JavaScript (opens in a new tab) (vendor measurement). Facts that appear only after scripts run may not be seen.
• Markup must match the page. No assistant's documentation reviewed here requires structured data for AI answers. Google asks that markup match the visible text (opens in a new tab) and says no special schema.org markup is needed (opens in a new tab). Bing says it "may support clearer grounding but does not guarantee visibility or grounding traffic." (opens in a new tab)
• The crawler documentation does not ask for AI text files. None of the OpenAI, Anthropic, or Perplexity crawler documents asks site owners to publish an llms.txt file; the OpenAI and Perplexity pages mention the format only to point to their own documentation indexes. Google says its search does not use AI text files (opens in a new tab).
• Manipulation is treated as spam. Google's spam policies cover attempts to manipulate generative AI responses, and define scaled content abuse regardless of how the content is created (opens in a new tab). Bing lists "Prompt Injection and AI Manipulation" as abuse, and states that "GEO does not guarantee grounding or citations." (opens in a new tab)

Access checklist by assistant

Each check follows from the documentation cited in the tables above. The checks are our recommendations.

Measuring across assistants

Only Google and Microsoft give site owners data about AI answers, as the tables show. For the rest, three sources fill part of the gap:

• Analytics. Google Analytics 4's AI Assistant channel counts visits "from sources like ChatGPT, Gemini, Deepseek, Copilot, or Grok" (opens in a new tab), and Microsoft Clarity labels referrals from Claude, Copilot, Gemini, ChatGPT, and Perplexity as AI platform traffic, though "If the source is hidden, the traffic might appear as Direct" (opens in a new tab). The AI referral traffic guide covers setup.
• Server logs. The platforms define ChatGPT-User, Claude-User, and Perplexity-User as fetching pages when users ask questions (see the tables above), so their visits indicate that a live answer opened a page. Visits from OAI-SearchBot, Claude-SearchBot, and PerplexityBot indicate indexing.
• Repeated prompt tests. Because answers vary, a panel of prompts run many times on each assistant gives a rate, not a rank.

Other AI crawlers in the same documentation

What the documentation does not settle

• ChatGPT and Bing. OpenAI names Microsoft as one search provider among others and runs its own crawler. How much of ChatGPT search comes from each source is not published.
• Claude and Brave. Anthropic lists Brave Search and TurboPuffer and runs Claude-SearchBot, but does not say how results are split between them.
• Perplexity and other indexes. Perplexity's documentation describes only its own crawlers.
• Google-Extended and AI Overviews. Google says the token does not affect Search, while one study found an association between blocking it and fewer AI Overview citations.
• Speed. No controlled study has measured how long a new page takes to appear in each assistant's answers.

Frequently asked questions

If I block GPTBot, will my site disappear from ChatGPT?

No. GPTBot collects training data, and OAI-SearchBot controls search. OpenAI says each setting is independent (opens in a new tab), so a site can block GPTBot and still appear in ChatGPT search answers.

Does blocking Google-Extended remove my pages from AI Overviews?

According to Google, no: Google-Extended "does not impact a site's inclusion in Google Search" (opens in a new tab), and AI Overviews use the Search index. It does control grounding in the Gemini app, and one study found that publishers blocking it were less likely to be retrieved by AI Overviews (opens in a new tab) (association only).

Does ChatGPT use Bing?

Partly. OpenAI names Microsoft among ChatGPT's search providers (opens in a new tab), and Enterprise and Edu accounts may send queries to Bing (opens in a new tab). ChatGPT also uses OpenAI's own index, built by OAI-SearchBot (opens in a new tab). Being indexed in Bing is one route into ChatGPT search, not the only one.

Can I see which of my pages AI assistants cite?

For Copilot, yes: Bing Webmaster Tools reports page-level citations (opens in a new tab). For AI Overviews and AI Mode, Search Console reports impressions (opens in a new tab). ChatGPT, Claude, and Perplexity document no citation report, so server logs, analytics, and repeated prompt tests are the options.

Why do assistants give different answers to the same question?

They search different indexes, and each answer is a sample. The ChatGPT and Gemini apps shared only 5.4% of cited domains for the same query (opens in a new tab) (preprint), and repeated prompts rarely return the same list of brands (opens in a new tab).

Next steps

To check whether these crawlers can reach your key pages, AEO HQ sells a fixed-price Instant AEO Audit for $499. It includes a live crawl of your key pages, AI crawler access checks, and a side-by-side analysis of up to three competitors.

Change log

• September 27, 2026: First published.