Google Tests Global AI Search Opt-Out Following UK Mandate and Cloudflare Deadline
Google is testing a tool that lets web publishers remove content from AI search answers without losing their position in Google Search results. The tech company plans to roll out this opt-out feature globally once trials finish in the United Kingdom.
This move follows direct regulatory enforcement in Britain and an upcoming deadline set by web infrastructure provider Cloudflare.
British Regulator Orders Search Choice
In June, the UK Competition and Markets Authority ordered Google to separate its search indexing from its AI content scraping. The antitrust agency ruled that Google must give site owners a clear choice to block their pages from AI features like AI Overviews.
Under the UK ruling, Google cannot lower the search rankings of websites that choose to opt out of AI features.
A Google spokesperson confirmed the ongoing test:
"The company is testing a new setting that lets websites opt out of its AI answers without hurting their search rankings, and will roll that out worldwide once UK testing is done."
Cloudflare Sets September 15 Enforcement Deadline
Cloudflare gave search and AI companies a September 15 deadline to unbundle their web crawlers. Starting on that date, Cloudflare will block mixed-purpose crawlers by default on ad-supported pages for new domains, new websites, and free-tier accounts.
The policy targets automated bots that combine general search indexing with AI model training and software agents. Cloudflare identified Google's main crawler, Googlebot, as a primary example of this bundled approach.
Cloudflare CEO Matthew Prince outlined the shift in web traffic driving the decision:
"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge."
Publisher Traffic Losses and Crawler Disparity
Automated bots now generate more than 50% of all internet traffic. Web publishers face declining referral traffic as AI-generated answers provide direct summaries on search pages.
Field studies show that Google's AI Overviews reduce outgoing clicks to source websites by about 40 per cent.
Publishers currently block independent AI crawlers like GPTBot and ClaudeBot at seven times the rate they block Googlebot. Fear of losing Google search referral traffic has prevented website owners from blocking Google's automated collection systems.
Payment Systems and Infrastructure Changes
Cloudflare is expanding a "Pay Per Use" system through technical agreements with AI search platforms You.com and Ceramic.ai. The service pays site owners when their published material shapes an AI-generated answer.
Cloudflare is also releasing tracking tools that show which automated crawlers collect publisher content and how much traffic those crawlers send back.
Google operates approximately 20 separate specialized web crawlers. The company maintains Googlebot as a combined crawler that indexes web pages for standard search and harvests content for AI models.
