Skip to content
AEO HQ

Tag

Crawling and indexing

Crawlers, robots.txt, and getting pages into the indexes AI assistants search.

29 of 29 articles

Answer engine optimization (AEO): the complete guide

What answer engine optimization (AEO) is, how AI answer engines choose sources, which tactics the research supports, and how to plan and measure AEO.

Complete guide · AEO

Sep 27, 2026
How to write content for answer engines

How to write AEO content that answer engines retrieve and cite: answer first, use the buyer's words, state checkable facts, and skip rewriting tricks.

Guide · AEO

Sep 28, 2026
AEO checklist

An AEO checklist of 50 checks for AI answers: crawler access, facts in HTML, content, entities, schema, third-party sources, offers, and measurement.

Checklist · AEO

Sep 28, 2026
AEO for HubSpot users

How HubSpot users can track answer engine visibility: what HubSpot's AEO tool measures, the AI Referrals source, what both leave out, and what to set up.

Use case · AEO

Sep 28, 2026
Generative engine optimization (GEO): the complete guide

Generative engine optimization (GEO), stage by stage: how AI engines retrieve and cite pages, which GEO strategies hold up in research, and how to test them.

Complete guide · GEO

Sep 28, 2026
GEO vs SEO: what is the difference?

GEO vs SEO: goals, where results appear, what is measured, and what drives inclusion. AI answers draw on search indexes, so the two overlap heavily.

Comparison · GEO

Sep 27, 2026
AI search optimization: SEO plus corroboration, facts, and measurement

What AI search optimization is, how AI assistants choose sources, which tactics hold up in research, and how to measure results. Every claim is sourced.

Complete guide · SEO+

Sep 27, 2026
How AI agents find and buy services

How AI agents research and buy services in 2026: what ChatGPT, Google, Copilot, Perplexity, Claude, and Stripe support, and what a service business can do.

Guide · SEO+

Sep 28, 2026
llms.txt: what it is and whether it matters

What llms.txt is, who reads it, and whether it helps AI visibility: the proposal, Google's position, server-log studies, and when a file is worth keeping.

Guide · SEO+

Sep 28, 2026
Technical SEO checklist for AI search

A technical SEO checklist for AI search: robots.txt, status codes, canonical URLs, redirects, rendering, page elements, and crawler identity, with sources.

Checklist · SEO+

Sep 28, 2026
SEO antipatterns

Twelve SEO mistakes that also keep pages out of AI answers, from blocked crawlers to facts hidden in JavaScript, with the evidence and a test for each.

Antipatterns · SEO+

Sep 27, 2026
How ChatGPT, Gemini, Claude, Perplexity, and Copilot find and cite sources

The search index, crawlers, inclusion controls, and site-owner reports behind ChatGPT, Gemini, Claude, Perplexity, and Copilot, from official docs.

Complete guide · AI assistants

Sep 27, 2026
ChatGPT SEO: how to rank in ChatGPT search

How ChatGPT search finds and cites pages: OpenAI's crawlers, Bing as a search provider, query rewriting, and the steps the evidence supports.

Guide · AI assistants

Sep 27, 2026
How to rank in Claude

How Claude's web search finds pages: Brave Search and TurboPuffer, what ClaudeBot, Claude-SearchBot, and Claude-User do, and what to allow.

Guide · AI assistants

Sep 27, 2026
How to rank in Gemini

How the Gemini app finds and cites pages: Google Search grounding, the Google-Extended token, what Google does not document, and what studies show.

Guide · AI assistants

Sep 28, 2026
How to rank in Google AI Overviews and AI Mode

What a page needs to appear in Google AI Overviews and AI Mode: indexing, snippet eligibility, one Search Console setting, and what studies show.

Guide · AI assistants

Sep 27, 2026
How to rank in Microsoft Copilot

How Microsoft Copilot finds and cites pages: Bing's index, the NOARCHIVE and NOCACHE controls, Bing's AI Performance report, and what studies show.

Guide · AI assistants

Sep 28, 2026
How to rank in Perplexity

How Perplexity finds and cites pages: what PerplexityBot and Perplexity-User do, what is not documented, and what citation studies show.

Guide · AI assistants

Sep 27, 2026
How AEO HQ measures AI visibility, and how our research is done

How AEO HQ grades evidence, pulls search data, and measures AI visibility, and what its $499 automated audit does and does not measure today.

Reference · Measurement

Sep 27, 2026
Answer engine optimization (AEO) services

Answer engine optimization services from AEO HQ: four fixed-price packages, $499 to $8,995, what each includes and excludes, how the work runs, and how to buy.

Service · Buying AEO

Sep 28, 2026
Instant AEO Audit

The Instant AEO Audit is AEO HQ's $499 automated audit of your site and up to three competitors: what it checks, what you receive, its limits, and how to buy.

Service · Buying AEO

Sep 28, 2026
AEO glossary: answer engine and AI search terms

Plain definitions of the terms used in answer engine optimization (AEO) and AI search, from AI Overviews to robots.txt, with a source for every fact.

Reference · Research

Sep 27, 2026
AI search statistics (2026)

Sourced AI search statistics for 2026: chatbot use, AI Overviews and clicks, AI referral traffic, cited sources, answer accuracy, and B2B buyers.

Reference · Research

Sep 28, 2026
SEO, AEO, and GEO for B2B SaaS companies

AEO for B2B SaaS: how software buyers use AI assistants, the sources they trust, and what SaaS companies should publish, fix, and measure.

Industry guide · Industries

Sep 27, 2026
SEO, AEO, and GEO for cybersecurity and IT services

SEO, AEO, and GEO for cybersecurity vendors and MSPs: what buyers are told to check, which sources AI cites, the rules on security claims, and what to measure.

Industry guide · Industries

Sep 28, 2026
SEO, AEO, and GEO for fintech and financial services

SEO, AEO, and GEO for fintech: how people use AI for money questions, which sources AI answers cite, the SEC, FINRA, CFPB, and FDIC rules, and what to measure.

Industry guide · Industries

Sep 28, 2026
SEO, AEO, and GEO for healthcare companies

AI search optimization for healthcare: how patients use AI to find care, which sources AI cites, the HIPAA and FTC rules that apply, and what to measure.

Industry guide · Industries

Sep 28, 2026
SEO, AEO, and GEO for law firms

Answer engine optimization for law firms: how clients use AI to find lawyers, which directories AI cites, the bar rules that apply, and what to measure.

Industry guide · Industries

Sep 27, 2026
SEO, AEO, and GEO for professional services firms

SEO, AEO, and GEO for accounting, tax, and consulting firms: how buyers use AI to find firms, what AI cites, the AICPA and IRS rules, and what to measure.

Industry guide · Industries

Sep 28, 2026

Next step

Find out who AI recommends.

Book the audit to see where you rank, where AI cites you, and where competitors win. The full audit price credits toward the Blueprint within 30 days.