Technical SEO checklist for AI search

A technical SEO checklist for AI search is a list of checks that confirm search crawlers can fetch a site's pages, read them without running scripts, index one version of each, and show them with a snippet. AI assistants find pages through the same crawlers and search indexes, so a page that fails these checks is harder, or impossible, for them to cite. This checklist has 23 checks in six groups, from the robots.txt file to the way crawlers are identified. Each check says how to verify it, what counts as a pass, how strong the evidence is, and which source explains it.

It belongs to AEO HQ's guide to AI search optimization, which treats the work as SEO plus corroboration, facts, and measurement. Google says its generative AI features on Google Search are "rooted in our core Search ranking and quality systems" (opens in a new tab), and Bing says Bing and Copilot "rely on the same core crawling, indexing, and ranking foundation as traditional search" (opens in a new tab). The AEO checklist already covers crawler access, indexing, and facts in the HTML. This page lists those checks in one table and adds the rest of technical SEO. The sources were read on September 27, 2026.

Scope

• Covers: crawling, rendering, indexing, and snippet eligibility for Google (including AI Overviews and AI Mode), Bing (including Copilot), and the crawlers of OpenAI, Anthropic, and Perplexity.
• Does not cover: content and wording, which are in the GEO checklist; names and profiles, in the entity and brand consistency checklist; sites in several languages; and crawl budget for very large sites. For very large and frequently updated sites, Google points owners to its guide to crawl budget (opens in a new tab).
• For: developers and technical marketers. Most checks need access to the server, the content delivery network (CDN), or the site's code.
• Pass criteria are AEO HQ's recommendations. The facts behind them link to their sources.
• Evidence labels:
  • Strong: official platform documentation, an Internet standard, or several independent studies that agree.
  • Moderate: laboratory studies, a single study, or one platform's guidance.
  • Weak: one small test, or a source with a conflict of interest.

Prerequisites

• A way to fetch a page without running JavaScript and with a chosen user agent, such as curl.
• Access to server or CDN logs, and to the CDN or firewall rules.
• Verified properties in Google Search Console and Bing Webmaster Tools.
• A list of every public host: the www and bare domain, subdomains, and any other host that serves pages.
• A site crawler that reports each URL's status code, title, meta description, canonical tag, and robots rules.

Checks already in the AEO checklist

Run these first. The first check of each AEO checklist group links to that group.

Quick checks to run first

If time is short, AEO HQ recommends starting with these five checks:

1. Check 1.1: every host serves its robots.txt file with a 200 status.
2. Check 1.3: robots.txt is not used to hide pages or protect content.
3. Check 3.1: each page resolves to one URL, on HTTPS and one host.
4. Check 4.1: links are <a> elements with an href, present in the server HTML.
5. Check 4.2: titles, canonical tags, and robots meta tags are in the server HTML.

1. The robots.txt file

robots.txt is a plain-text file at the root of a host that tells crawlers which URLs they may fetch.

• 1.1 Every host serves its robots.txt file with a 200 status. Verify: for each host and protocol, run curl -sI https://www.example.com/robots.txt. Pass: each returns HTTP 200 with a text/plain content type, or 404 where a host needs no rules. None returns a 5xx error. Why: under the Internet standard, the file must be at "/robots.txt" in the top-level path of each host, "UTF-8 encoded," with the media type "text/plain" (opens in a new tab). If the file is "unreachable due to server or network errors," a crawler "MUST assume complete disallow," while a 4xx status lets it "access any resources on the server" (opens in a new tab). A server error on robots.txt can therefore stop compliant crawlers from fetching anything on the host. Evidence: strong (Internet standard).
• 1.2 robots.txt names the sitemap. Verify: read the file. Pass: it has a Sitemap: line with the absolute URL of each sitemap or sitemap index. Why: Google says to add a Sitemap: line "anywhere in your robots.txt file," and "We will find it the next time we crawl your robots.txt file" (opens in a new tab). The standard lets crawlers interpret records that are not part of the protocol, such as "Sitemaps" (opens in a new tab). Evidence: strong (official documentation and the standard).
• 1.3 robots.txt is not used to hide pages or protect content. Verify: list every Disallow rule and write down what it is for. Pass: pages that must stay out of search results carry noindex and can be crawled, private content requires a login, and the file lists no path that should stay private. Why: Google says robots.txt "is not a mechanism for keeping a web page out of Google" (opens in a new tab), and that a disallowed URL can still be indexed if it "is linked from other places on the web" (opens in a new tab). If a page is blocked by robots.txt, "the crawler will never see the noindex rule" (opens in a new tab). OpenAI says it may surface just the link and title of a disallowed page in ChatGPT Atlas, and that its crawler must be allowed to crawl a page to read a noindex tag (opens in a new tab). The standard says its rules "are not a form of access authorization" (opens in a new tab) and that listing paths in the file "exposes them publicly" (opens in a new tab). Fetches that a person triggers may not follow it: OpenAI says of its ChatGPT-User agent that "robots.txt rules may not apply" (opens in a new tab), and Perplexity says its Perplexity-User fetcher "generally ignores robots.txt rules" (opens in a new tab). Evidence: strong (official documentation and the standard).

2. Server responses

• 2.1 Key pages and navigation links return 200 without a redirect. Verify: request each key URL, and each URL linked from the navigation, without following redirects: curl -s -o /dev/null -w "%{http_code}" URL. Pass: every one returns 200. Why: Google's technical requirements say a page must work, meaning "that Google receives an HTTP 200 (success) status code" (opens in a new tab). Linking to the final URL also avoids redirect hops, which check 3.4 covers. Evidence: strong (official documentation).
• 2.2 Removed pages return 404 or 410. Verify: request a URL that never existed, and URLs removed in the past year. Pass: each returns 404 or 410, not a 200 page that says "not found," and removed URLs are sent through IndexNow (the AEO checklist's check 1.5). Why: Google says it doesn't index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code "are removed from the index" (opens in a new tab), and it reports pages that show an error but return a success status as soft 404 errors in Search Console (opens in a new tab). For single-page apps, Google suggests that you "Use a JavaScript redirect to a URL for which the server responds with a 404 HTTP status code" (opens in a new tab) or add a noindex tag to error pages. Bing asks site owners, when content is permanently removed, to "Return a 404 status code" (opens in a new tab). Evidence: strong (official documentation).
• 2.3 Crawlers rarely receive 5xx or 429 responses. Verify: filter server logs for requests from verified crawlers, and count 5xx and 429 responses by day. Pass: no run of 5xx or 429 responses to crawlers lasts beyond a single incident, and rate limits do not throttle verified crawlers at their normal rate. Why: Google says "5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling" (opens in a new tab), and that it treats 429 "as a signal that the server is overloaded" (opens in a new tab). Bing says it "allocates crawl capacity based on site health, efficiency, signal quality, and crawl value" (opens in a new tab). Evidence: strong for Google and Bing; we found no statement on how the AI companies' crawlers handle these codes.
• 2.4 Pages send validation headers that change only with the content. Verify: request a page twice with curl -sI and read the ETag and Last-Modified headers. Pass: at least one is present, it is the same across requests, and it changes when the content changes. Why: Bing says "freshness signals like accurate lastmod values and HTTP validation headers such as ETags help Bing detect content changes more reliably" (opens in a new tab). Evidence: moderate (one platform's guidance); no study has measured the effect on AI answers.

3. One URL for each page

• 3.1 Each page resolves to one URL, on HTTPS and one host. Verify: request the http and https versions of key pages, with and without www, and with and without a trailing slash. Pass: every variant reaches the same URL through one permanent redirect. Why: Google says it "prefers HTTPS pages over equivalent HTTP pages as canonical, except when there are issues or conflicting signals" (opens in a new tab), and that a redirect is "A strong signal that the target of the redirect should become canonical" (opens in a new tab). Bing says "Duplicate URLs dilute signals and reduce Bing's confidence in selecting a URL for grounding results or citations" (opens in a new tab). Grounding means basing an answer on retrieved sources. Evidence: strong (official documentation).
• 3.2 Each indexable page names itself as canonical, in the server HTML. Verify: view the raw HTML of each page template and read its rel="canonical" link. Pass: it gives the page's own absolute URL, and the value is the same after scripts run. Why: Google calls rel="canonical" "A strong signal that the specified URL should become canonical" (opens in a new tab), recommends absolute URLs in it, and says: "Don't specify different URLs as canonical for the same page using different canonicalization techniques" (opens in a new tab). For pages built with JavaScript, it says to "make sure that you always set the canonical URL to the same value as the original HTML" (opens in a new tab). Evidence: strong (official documentation).
• 3.3 URLs with tracking parameters point to the clean URL. Verify: open a key page with ?utm_source=chatgpt.com added to its URL and read its canonical tag, and search internal links for parameters. Pass: the canonical tag gives the URL without the parameter, and internal links use clean URLs. Why: ChatGPT "automatically includes the UTM parameter utm_source=chatgpt.com in referral URLs" (opens in a new tab). Bing asks site owners to "Use canonical URLs, parameter controls, and consistent URL structures to consolidate signals" (opens in a new tab), and Google says: "When linking within your site, link to the canonical URL rather than a duplicate URL" (opens in a new tab). Evidence: strong (official documentation).
• 3.4 Moved pages use one permanent server-side redirect. Verify: collect old URLs from past sitemaps, analytics, and backlink reports, and request each. Pass: each old URL returns 301 or 308 straight to the page that replaced it, with no chain of redirects and no temporary redirect for a permanent move. Why: Google says: "We recommend that you use a permanent server-side redirect whenever possible" (opens in a new tab). With a permanent redirect, Google shows the new target in search results; with a temporary one, it may keep the old URL (opens in a new tab). Google's crawlers follow up to 10 redirect hops (opens in a new tab). Bing asks for "301 redirects for permanent URL changes" (opens in a new tab), 302 redirects "only for very short-term changes (less than 2 days)" (opens in a new tab), and "redirects instead of canonical tags" (opens in a new tab) when content moves. Evidence: strong (official documentation).
• 3.5 The sitemap lists only canonical, indexable URLs. Verify: fetch each sitemap and compare every URL with its status code, canonical tag, and robots meta tag. Pass: all URLs are absolute; each returns 200, names itself as canonical, and has no noindex; and each file holds fewer than 50,000 URLs and 50 MB. Submission and lastmod are the AEO checklist's check 1.4. Why: Google asks for "fully-qualified, absolute URLs" (opens in a new tab), says "All formats limit a single sitemap to 50MB (uncompressed) or 50,000 URLs" (opens in a new tab), and asks owners who have the same content under several URLs to choose one "and include that in the sitemap instead of all URLs that lead to the same content" (opens in a new tab). Bing asks sitemaps to "List only canonical URLs" (opens in a new tab) and to "Remove deleted or redirected URLs promptly" (opens in a new tab). Evidence: strong (official documentation).

4. Links and rendering

• 4.1 Links are <a> elements with an href, present in the server HTML. Verify: view the raw HTML of the home page and a hub page, and check the navigation, footer, and in-text links. Pass: each is an <a href> to a real URL, without JavaScript; anchor text describes the page it links to; and single-page apps route between views with real URLs, not # fragments. Why: Google says "Google can only crawl your link if it's an <a> HTML element (also known as anchor element) with an href attribute" (opens in a new tab) and asks for anchor text that is "descriptive, reasonably concise, and relevant to the page that it's on and to the page it links to" (opens in a new tab). Bing asks for "standard <a href> links" (opens in a new tab). For single-page apps, Google says to "use the History API to implement routing between different views" (opens in a new tab). A link that exists only after scripts run is invisible to crawlers that do not run them: in Vercel's network data from December 2024, none of the major AI crawlers rendered JavaScript (opens in a new tab). Evidence: strong.
• 4.2 Titles, canonical tags, and robots meta tags are in the server HTML, and scripts do not change them. Verify: compare the raw HTML with the rendered page, for example with URL Inspection in Search Console, which Google says shows "the HTML that Googlebot received while crawling the page" (opens in a new tab). Pass: the <title>, meta description, canonical link, and robots meta tag are the same in both. Why: Google says "When Google encounters the noindex tag, it may skip rendering and JavaScript execution" (opens in a new tab), and that "server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript" (opens in a new tab). Evidence: strong (official documentation).
• 4.3 Scripts and styles that pages need are not blocked. Verify: read robots.txt for rules that block script, style, or image folders, and use URL Inspection to view the rendered page. Pass: no file needed to render a page is disallowed for Googlebot or Bingbot. Why: Google says it "is able to process content within JavaScript as long as it isn't blocked" (opens in a new tab). Bing asks site owners to avoid "Blocking important URLs unnecessarily" (opens in a new tab) and "Excessive or unnecessary HTTP requests to render the content" (opens in a new tab). Evidence: strong (official documentation).

5. Page elements

• 5.1 Each indexable page has its own title and meta description. Verify: export titles and meta descriptions from a crawl and sort them. Pass: none is missing, duplicated, or only a few words long. Wording is the AEO checklist's check 3.2. Why: Bing says "Missing, duplicate, or overly short title tags and meta descriptions may reduce indexing reliability, ranking, and eligibility for grounding results and citations" (opens in a new tab). Google says indexing includes processing "key content tags and attributes, such as <title> elements and alt attributes" (opens in a new tab). In a laboratory pipeline, extending optimization from body text alone to titles, meta descriptions, headings, and schema fields raised the retrieval hit rate by 22% (opens in a new tab) (peer-reviewed). Evidence: strong for Bing's guidance; moderate for the effect (one laboratory study).
• 5.2 Headings form one logical outline. Verify: list the H1 to H3 headings of each page template. Pass: one H1 names the page's topic, and H2 and H3 headings follow in order without skipped levels. Why: Bing lists a "Logical <H1> – <H6> heading hierarchy" (opens in a new tab) among its practices for clear structure. Google says perfectly semantic HTML is not required, but "it's generally a good idea to try to use semantic HTML when possible" (opens in a new tab). Evidence: moderate (platform guidance).
• 5.3 Images have alt text, and videos have captions or transcripts. Verify: crawl for images with no alt attribute, and list the videos on key pages. Pass: every image that carries information has alt text that describes it, and every video that carries information has captions or a transcript on the page. Why: Bing says images and video "should not be the sole source of information required to understand the topic" (opens in a new tab), and asks for descriptive file names, alt text, and "Captions, transcripts, or structured data" (opens in a new tab). Evidence: strong (official documentation).
• 5.4 The mobile page has the same content as the desktop page. Verify: load key pages at a phone's width and compare them with the desktop version: main text, headings, title, meta description, robots meta tag, and structured data. Pass: all match, and no primary content waits for a tap or a swipe to load. Why: Google says it "uses the mobile version of a site's content, crawled with the smartphone agent, for indexing and ranking" (opens in a new tab). It asks site owners to "Make sure that your mobile site contains the same content as your desktop site" (opens in a new tab) and to "Use the same robots meta tags on the mobile and desktop site" (opens in a new tab), and says: "Don't lazy-load primary content upon user interaction" (opens in a new tab). Evidence: strong (official documentation).
• 5.5 Buttons, forms, and menus have accessible names. Verify: run an accessibility checker on key pages, including the pricing and checkout pages. Pass: every button, link, form field, and menu has a text label or an ARIA label that says what it does. Why: OpenAI says ChatGPT Atlas uses "ARIA tags—the same labels and roles that support screen readers—to interpret page structure and interactive elements" (opens in a new tab). Google says browser agents may work by "analyzing visual renderings (like screenshots), inspecting the DOM structure, and interpreting the accessibility tree" (opens in a new tab). Evidence: strong for the documentation; the effect on AI recommendations has not been measured. See how AI agents find and buy services.
• 5.6 Pages that fail Core Web Vitals badly are fixed, and passing pages are left alone. Verify: open the Core Web Vitals report in Search Console. Pass: no key page is rated "Poor," and further speed work on passing pages is not listed as an AI visibility task. Why: Google lists "Providing a great page experience for users" (opens in a new tab) among the SEO basics for its AI features. Among 107,352 pages already visible in AI Overviews and AI Mode, the correlation between Largest Contentful Paint and AI visibility ranged from -0.12 to -0.18 (opens in a new tab), and the analyst concluded: "Good performance does not create an advantage. Severe failure creates disadvantage." (opens in a new tab) (practitioner analysis, January 2026; only pages that were already visible). SEO antipatterns covers the cost of over-investing here. Evidence: weak (one correlational analysis).

6. Crawler identity

• 6.1 Crawlers get the same content as people. Verify: fetch key pages with a browser user agent and with the user agents of Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, and PerplexityBot, and compare the main text. Review server and CDN rules that vary content by user agent. Pass: the main content is the same for every user agent, and no version of a page is served only to crawlers or only to AI crawlers. Why: Google defines cloaking as "the practice of presenting different content to users and search engines with the intent to manipulate search rankings and mislead users" (opens in a new tab), including "Inserting text or keywords into a page only when the user agent that is requesting the page is a search engine, not a human visitor" (opens in a new tab). Bing says "Sites should not rely on cloaking or serving materially different experiences to crawlers and visitors" (opens in a new tab). Evidence: strong (official policies).
• 6.2 Firewall rules recognize crawlers by IP range or signature, not by user agent alone. Verify: read the CDN and firewall rules that allow or block bots. Pass: each rule that lets a crawler through also checks the crawler's published IP ranges or a verified-bot category, and signed agents are checked by their signatures. Why: Google warns that "The HTTP user agent string can be spoofed" (opens in a new tab) and publishes the IP ranges and reverse DNS names its common crawlers use (opens in a new tab). Perplexity's firewall instructions say to set up a rule that "combines both User-Agent and IP address conditions" (opens in a new tab). OpenAI's cloud browser uses "Web Bot Auth to sign outbound HTTP requests, allowing website operators to verify that requests genuinely originate from ChatGPT" (opens in a new tab). Evidence: strong (official documentation).

Version and change log

• Version 1.0, September 28, 2026: First published. Sources checked on September 27, 2026.

Next steps

AEO HQ's Instant AEO Audit ($499) automates a few of these checks for a site's home page: it requests robots.txt and sitemap.xml, records whether robots.txt lists a sitemap, and records the home page's canonical URL and meta robots tag. Its limits are listed in our methodology.