Introducing pay per crawl: Enabling content owners to charge AI crawlers for access

submitted by punrca to webdev 9 points | 2 comments https://blog.cloudflare.com/introducing-pay-per-crawl/ > Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content. > Pay per crawl grants domain owners full control over their monetization strategy. They can define a flat, per-request price across their entire site. Publishers will then have three distinct options for a crawler: 1. Allow: Grant the crawler free access to content. 2. Charge: Require payment at the configured, domain-wide price. 3. Block: Deny access entirely, with no option to pay. Although this is old news, I still found it interesting. Also, I like the “AI Labyrinth” feature of Cloudflare to block AI bots.