Tag
Crawling and indexing
Crawlers, robots.txt, and getting pages into the indexes AI assistants search.
29 of 29 articles
| Answer engine optimization (AEO): the complete guide What answer engine optimization (AEO) is, how AI answer engines choose sources, which tactics the research supports, and how to plan and measure AEO. Complete guide · AEO | Complete guide | AEO | Sep 27, 2026 |
| How to write content for answer engines How to write AEO content that answer engines retrieve and cite: answer first, use the buyer's words, state checkable facts, and skip rewriting tricks. Guide · AEO | Guide | AEO | Sep 28, 2026 |
| AEO checklist An AEO checklist of 50 checks for AI answers: crawler access, facts in HTML, content, entities, schema, third-party sources, offers, and measurement. Checklist · AEO | Checklist | AEO | Sep 28, 2026 |
| AEO for HubSpot users How HubSpot users can track answer engine visibility: what HubSpot's AEO tool measures, the AI Referrals source, what both leave out, and what to set up. Use case · AEO | Use case | AEO | Sep 28, 2026 |
| Generative engine optimization (GEO): the complete guide Generative engine optimization (GEO), stage by stage: how AI engines retrieve and cite pages, which GEO strategies hold up in research, and how to test them. Complete guide · GEO | Complete guide | GEO | Sep 28, 2026 |
| GEO vs SEO: what is the difference? GEO vs SEO: goals, where results appear, what is measured, and what drives inclusion. AI answers draw on search indexes, so the two overlap heavily. Comparison · GEO | Comparison | GEO | Sep 27, 2026 |
| AI search optimization: SEO plus corroboration, facts, and measurement What AI search optimization is, how AI assistants choose sources, which tactics hold up in research, and how to measure results. Every claim is sourced. Complete guide · SEO+ | Complete guide | SEO+ | Sep 27, 2026 |
| How AI agents find and buy services How AI agents research and buy services in 2026: what ChatGPT, Google, Copilot, Perplexity, Claude, and Stripe support, and what a service business can do. Guide · SEO+ | Guide | SEO+ | Sep 28, 2026 |
| llms.txt: what it is and whether it matters What llms.txt is, who reads it, and whether it helps AI visibility: the proposal, Google's position, server-log studies, and when a file is worth keeping. Guide · SEO+ | Guide | SEO+ | Sep 28, 2026 |
| Technical SEO checklist for AI search A technical SEO checklist for AI search: robots.txt, status codes, canonical URLs, redirects, rendering, page elements, and crawler identity, with sources. Checklist · SEO+ | Checklist | SEO+ | Sep 28, 2026 |
| SEO antipatterns Twelve SEO mistakes that also keep pages out of AI answers, from blocked crawlers to facts hidden in JavaScript, with the evidence and a test for each. Antipatterns · SEO+ | Antipatterns | SEO+ | Sep 27, 2026 |
| How ChatGPT, Gemini, Claude, Perplexity, and Copilot find and cite sources The search index, crawlers, inclusion controls, and site-owner reports behind ChatGPT, Gemini, Claude, Perplexity, and Copilot, from official docs. Complete guide · AI assistants | Complete guide | AI assistants | Sep 27, 2026 |
| ChatGPT SEO: how to rank in ChatGPT search How ChatGPT search finds and cites pages: OpenAI's crawlers, Bing as a search provider, query rewriting, and the steps the evidence supports. Guide · AI assistants | Guide | AI assistants | Sep 27, 2026 |
| How to rank in Claude How Claude's web search finds pages: Brave Search and TurboPuffer, what ClaudeBot, Claude-SearchBot, and Claude-User do, and what to allow. Guide · AI assistants | Guide | AI assistants | Sep 27, 2026 |
| How to rank in Gemini How the Gemini app finds and cites pages: Google Search grounding, the Google-Extended token, what Google does not document, and what studies show. Guide · AI assistants | Guide | AI assistants | Sep 28, 2026 |
| How to rank in Google AI Overviews and AI Mode What a page needs to appear in Google AI Overviews and AI Mode: indexing, snippet eligibility, one Search Console setting, and what studies show. Guide · AI assistants | Guide | AI assistants | Sep 27, 2026 |
| How to rank in Microsoft Copilot How Microsoft Copilot finds and cites pages: Bing's index, the NOARCHIVE and NOCACHE controls, Bing's AI Performance report, and what studies show. Guide · AI assistants | Guide | AI assistants | Sep 28, 2026 |
| How to rank in Perplexity How Perplexity finds and cites pages: what PerplexityBot and Perplexity-User do, what is not documented, and what citation studies show. Guide · AI assistants | Guide | AI assistants | Sep 27, 2026 |
| How AEO HQ measures AI visibility, and how our research is done How AEO HQ grades evidence, pulls search data, and measures AI visibility, and what its $499 automated audit does and does not measure today. Reference · Measurement | Reference | Measurement | Sep 27, 2026 |
| Answer engine optimization (AEO) services Answer engine optimization services from AEO HQ: four fixed-price packages, $499 to $8,995, what each includes and excludes, how the work runs, and how to buy. Service · Buying AEO | Service | Buying AEO | Sep 28, 2026 |
| Instant AEO Audit The Instant AEO Audit is AEO HQ's $499 automated audit of your site and up to three competitors: what it checks, what you receive, its limits, and how to buy. Service · Buying AEO | Service | Buying AEO | Sep 28, 2026 |
| AEO glossary: answer engine and AI search terms Plain definitions of the terms used in answer engine optimization (AEO) and AI search, from AI Overviews to robots.txt, with a source for every fact. Reference · Research | Reference | Research | Sep 27, 2026 |
| AI search statistics (2026) Sourced AI search statistics for 2026: chatbot use, AI Overviews and clicks, AI referral traffic, cited sources, answer accuracy, and B2B buyers. Reference · Research | Reference | Research | Sep 28, 2026 |
| SEO, AEO, and GEO for B2B SaaS companies AEO for B2B SaaS: how software buyers use AI assistants, the sources they trust, and what SaaS companies should publish, fix, and measure. Industry guide · Industries | Industry guide | Industries | Sep 27, 2026 |
| SEO, AEO, and GEO for cybersecurity and IT services SEO, AEO, and GEO for cybersecurity vendors and MSPs: what buyers are told to check, which sources AI cites, the rules on security claims, and what to measure. Industry guide · Industries | Industry guide | Industries | Sep 28, 2026 |
| SEO, AEO, and GEO for fintech and financial services SEO, AEO, and GEO for fintech: how people use AI for money questions, which sources AI answers cite, the SEC, FINRA, CFPB, and FDIC rules, and what to measure. Industry guide · Industries | Industry guide | Industries | Sep 28, 2026 |
| SEO, AEO, and GEO for healthcare companies AI search optimization for healthcare: how patients use AI to find care, which sources AI cites, the HIPAA and FTC rules that apply, and what to measure. Industry guide · Industries | Industry guide | Industries | Sep 28, 2026 |
| SEO, AEO, and GEO for law firms Answer engine optimization for law firms: how clients use AI to find lawyers, which directories AI cites, the bar rules that apply, and what to measure. Industry guide · Industries | Industry guide | Industries | Sep 27, 2026 |
| SEO, AEO, and GEO for professional services firms SEO, AEO, and GEO for accounting, tax, and consulting firms: how buyers use AI to find firms, what AI cites, the AICPA and IRS rules, and what to measure. Industry guide · Industries | Industry guide | Industries | Sep 28, 2026 |
Next step
Find out who AI recommends.
Book the audit to see where you rank, where AI cites you, and where competitors win. The full audit price credits toward the Blueprint within 30 days.