ClaudeBot and PerplexityBot: how they crawl and how to track them

By Stas Levitan, CEO · May 8, 2026 · 5 min read

ClaudeBot vs PerplexityBot: two crawlers, two intents

ClaudeBot and PerplexityBot both show up in server logs, but they are there for very different reasons. ClaudeBot is Anthropic's training and retrieval crawler; PerplexityBot is closer to a live search-engine crawler that fetches pages in response to individual user questions. Knowing which is which — and separating them from spoofed traffic — is the first step toward tracking AI crawler activity in a way that actually informs strategy rather than just adding another dashboard.

ClaudeBot is Anthropic's training and retrieval crawler. Its presence on your site means Anthropic considers your content useful enough to ingest. The fetch pattern is patient and broad — Claude will revisit URLs over weeks, often without a tight correlation to a single user query. PerplexityBot is much closer to a live search engine crawler. It fetches in response to user questions. When you see PerplexityBot hit /pricing three times in a minute, it's very likely a user just asked Perplexity about your pricing. The latency from crawl to citation is short.

User agents to watch

ClaudeBot/1.0 (+claudebot@anthropic.com)
PerplexityBot/1.0 (+https://docs.perplexity.ai/docs/perplexitybot)
Perplexity-User/1.0

Perplexity-User is the on-demand fetch initiated by a single user query. It's the closest thing to a "real-time citation" signal you'll get without instrumenting the AI engine itself.

What healthy traffic looks like

For a mid-market SaaS site running proper GEO:

ClaudeBot: 50–500 hits/week across the site, weighted toward documentation and blog.
PerplexityBot: spiky, correlated with topical news cycles and product launches.
Perplexity-User: low volume but high signal — each one is an actual answer being assembled.

If ClaudeBot drops to zero for two weeks straight, something on your robots.txt or sitemap regressed. If PerplexityBot stops touching your pricing page after a redesign, your schema probably broke and Perplexity can't extract structured offers anymore.

Why log filtering isn't enough

User-agent strings are trivially spoofable. A real platform verifies bots against published IP ranges:

Anthropic publishes claudebot.txt listing valid CIDR blocks.
Perplexity publishes its ranges via DNS and a docs page.
OpenAI does the same for GPTBot and OAI-SearchBot.

Without verification you'll over-count by 10–30%, mostly from scrapers and competitive intelligence tools impersonating ClaudeBot.

Turning traffic into action

Three workflows that change once you can see this data:

Page prioritization. If PerplexityBot is hammering /integrations but ignoring /security, you know which page needs schema first.

Deploy guardrails. Crawl-drop alerts catch the noindex accident a week before GSC notices.

Sales enablement. "Last month, AI engines fetched our pricing page 1,200 times" is a board-deck line that didn't exist two years ago.

Frequently asked

Does Claude search the live web? Claude has both training data and live retrieval depending on the product. ClaudeBot covers the training side; live retrieval is browser-based. Should I block PerplexityBot? Almost never. Blocking it removes you from one of the fastest-growing answer engines. How do I separate Perplexity-User from PerplexityBot? Different user-agent strings; treat them as two distinct event types in your analytics.