Google-Agent Arrives: Robots.txt Gets a Smaller Crown

What happened: Google quietly added "Google-Agent" to its list of user-triggered web fetchers, and a No Hacks explainer says it’s the user agent for AI systems on Google infrastructure that browse websites on behalf of users (with Project Mariner named as the first product using it).

Why it matters: The big twist is access control: Google classifies Google-Agent as a user-triggered fetcher and says those fetchers "generally ignore robots.txt rules" — meaning the polite ‘please don’t crawl me’ sign isn’t the same as a lock anymore.

Wider context: The article contrasts this with OpenAI’s ChatGPT-User and Anthropic’s Claude-User traffic, which it says respect robots.txt directives. If your strategy was ‘block the bots,’ the agent era is turning that into ‘build real gates.’

Background: No Hacks highlights Google experimenting with the IETF draft "web-bot-auth" protocol (identity: https://agent.bot.goog), which uses cryptographic signing so websites can verify an agent’s identity — because user-agent strings are cosplay.


Singularity Soup Take: The web is graduating from "robots.txt vibes" to "cryptographic paperwork." If agents are going to click buttons and fill forms for humans, the only sane future is verified identity — and a lot of sites discovering they never built real access control.

Key Takeaways:

  • New visitor class: The explainer describes Google-Agent as a user-triggered fetcher for AI agents browsing on behalf of users, distinct from Googlebot’s continuous indexing, and ties it to Google’s experimental Project Mariner browsing tool.
  • Robots.txt gap: It says Google’s position is that user-triggered fetchers generally ignore robots.txt because the request is initiated by a human — so blocking Google-Agent requires the same kinds of server-side authentication and access controls you’d use on real people.
  • Signed agent identity: The piece notes Google is experimenting with web-bot-auth using the identity https://agent.bot.goog, where agents cryptographically sign HTTP requests so websites can verify ‘who is visiting,’ reducing spoofing compared to plain user-agent strings.