Blocking AI Crawlers Costs Publishers Real Human Traffic

Blocking AI Crawlers Costs Publishers Real Human Traffic

What happened: PPC Land highlights an updated working paper from Rutgers and Wharton arguing that news publishers who blocked major LLM crawlers using robots.txt saw roughly a 7% decline in weekly visits within six weeks of blocking.

Why it matters: The paper’s uncomfortable claim is that blocking isn’t a neat ‘protect our content’ button — it’s a ‘reduce our reach’ button, with traffic declines showing up in human browsing panel data, not just in server logs full of bots behaving badly.

Wider context: The study describes a staggered wave of publisher blocks across multiple AI crawlers (including OpenAI, Anthropic, Perplexity, Google-Extended, and others) and treats the rise of AI answer engines as a distribution channel that can shift brand exposure.

Background: The working paper draws on sources including SimilarWeb, Semrush, Comscore’s web-behavior panel, HTTP Archive robots.txt snapshots, and other datasets, and focuses its main causal window on the period before Google’s May 2024 AI Overviews rollout.

Blocking AI crawlers cost news publishers 7% of traffic, study finds — PPC Land

Singularity Soup Take: Robots.txt was built for polite search engines, not for a world where every model wants to be your homepage — so publishers are discovering the fun new trade: block the machines, and you might also block the humans who follow them.

Key Takeaways:

  • Measured hit: PPC Land reports the paper estimates an ~7% drop in weekly visits within six weeks after blocking, with similar negative estimates across multiple traffic datasets and methods — and with one dataset tracking actual household browsing behavior.
  • Who got blocked: The article lists common targets such as GPTBot and ChatGPT-User (OpenAI), ClaudeBot and Claude-User (Anthropic), PerplexityBot, Google-Extended, and ByteSpider, reflecting how publishers tried to wall off both training and retrieval crawlers.
  • Distribution as the mechanism: The paper’s proposed channel is reduced exposure and referrals from AI-generated answers and summaries, where being ‘missing’ from machine citations can translate into fewer direct visits over time — even if that feels like the opposite of fair.

Relevant Resources

AI and Society: How Technology Will Change Our World — A broader view of how AI shifts incentives and gatekeepers, including who gets seen, cited, and visited online.

Details