Cloudflare Lets Sites Charge AI for Data Crawling

Cloudflare has introduced a new feature that lets website owners control access to their data by artificial intelligence systems. The company announced Tuesday that Cloudflare lets sites charge or block web crawlers used by AI companies to scrape online content. This move targets the growing demand for publicly available data to train large language models, which often gather information without permission.

The feature offers website operators the ability to set rules for AI bots, either denying access outright or requiring payment before data can be collected. Cloudflare lets sites charge based on usage, helping businesses preserve bandwidth and protect intellectual property. The rollout comes amid rising concerns over how generative AI systems acquire and utilize online content.

Cloudflare’s update empowers customers to manage how their data is used in AI development. Read more about the company’s new “pay-per-crawl” feature and its implications here:

Cloudflare rolls out ‘pay-per-crawl’ feature to constrain AI’s limitless hunger for data