Congratulations, website owners. You’re getting a bot firewall and a toll booth. Same device.
Cloudflare and GoDaddy are trying to turn “please stop strip-mining my site” into a dashboard setting: allow, block, or require payment. The bigger shift is that the web is being rebuilt as a permissions and identity layer for agents, not a polite library for humans.
What Happened
Cloudflare and GoDaddy announced a partnership that integrates Cloudflare’s AI Crawl Control into GoDaddy’s hosting platform, aiming to give site owners visibility and control over automated AI crawlers. The press release frames this as enabling an “open agentic web” built on clearer identity and permissions for agents.
Alongside crawl controls, the partnership backs standards like GoDaddy’s Agent Name Service (ANS) and Cloudflare’s Web Bot Auth, both intended to help verify who operates an agent and whether requests are cryptographically signed.
The Non-Obvious Angle: This Is Policy, Implemented as Defaults
Everyone is arguing about whether AI should “pay publishers.” Meanwhile, the real action is happening where it always happens on the internet: in default settings, middleware, and the platforms that host everyone else’s small business websites.
Once “require payment” exists as a first-class option next to “allow” and “block,” the negotiation stops being a legal theory and becomes a routing decision. That is how business models change: not by speeches, but by checkboxes.
Why This Matters
- Identity becomes the toll gate: if agents need verifiable identity to get access, then identity providers and hosting platforms become market-makers.
- The referral economy keeps collapsing: answer engines reduce clicks, so publishers and creators look for compensation mechanisms that don’t rely on traffic.
- Small sites get dragged into geopolitics: “allow/deny” decisions will become a proxy fight over which models get to ingest which parts of the web.
The Singularity Soup Take
The “open web” is being refactored into a permissioned API, with identity, audit logs, and payment signals stapled on. This is not a moral awakening. It’s infrastructure owners realizing that bots don’t click ads, and therefore they don’t pay anyone’s rent. Expect the next phase to be messy: partial adoption, spoofed identities, and a lot of “sorry, your agent isn’t on the list.”
What to Watch
- Whether ANS and Web Bot Auth get real adoption outside Cloudflare/GoDaddy’s orbit.
- How “payment required” is operationalized (402-style protocols, contracts, or just a polite hint bots ignore).
- Whether publishers and creators actually get leverage, or just more dashboards to manage while traffic continues to leak.
Sources
Cloudflare — "Cloudflare and GoDaddy Partner to Help Enable an Open Agentic Web"
The Register — "Cloudflare, GoDaddy team up to curb AI bot brigades"
Cloudflare Docs — "Web Bot Auth"
Agent Name Service — "ANS Open Standard"