Here’s some good news for website owners globally. Cloudflare has expanded its tools for managing artificial intelligence (AI) web crawlers, giving website owners more control over how AI companies access their content as concerns grow over declining referral traffic and content monetization. The announcement marks the company’s second annual “Content Independence Day” initiative.
The company said its previous approach, which allowed customers to block AI bots with a single setting, did not provide enough flexibility.
Instead, website owners can now distinguish between three categories of AI traffic: Search crawlers that index content for search engines, Agent crawlers that retrieve information on behalf of AI assistants, and Training crawlers that collect data for developing AI models. Customers can choose which categories to allow or block, including those using Cloudflare’s free tier.
Cloudflare also announced new default settings that will take effect on September 15, 2026. For new domains, Training and Agent crawlers will be blocked by default on pages displaying advertisements, while Search crawlers will remain allowed. The company argues that ad-supported pages are designed for human visitors and that unrestricted AI access could undermine publisher revenue. Existing customers will be able to opt out of the new defaults before they are applied.
Another significant change targets AI companies that use a single crawler for multiple purposes. Under the updated system, multi-purpose crawlers will be evaluated based on all of their activities. This means a crawler used for both search indexing and AI training could be blocked if a website owner chooses to restrict training access. Cloudflare said it encourages AI companies to separate their crawlers to improve transparency for publishers.
For enterprise customers, Cloudflare also introduced BotBase, a searchable database of verified bots and automated agents designed to improve visibility into automated traffic. The company said the new controls reflect a broader effort to help publishers balance AI innovation with sustainable business models for online content.