Creative Commons Supports AI ‘Pay-to-Crawl’ Systems for Fair Compensation
However, the nonprofit organization Creative Commons has taken a cautious approach to the concept, expressing its support for pay-to-crawl technology.
Meanwhile, the organization has been working on a framework for an open AI ecosystem, which includes a plan for dataset sharing between companies and AI providers.
Consequently, the nonprofit is now backing pay-to-crawl systems, which aim to automate compensation for website content accessed by machines, like AI web crawlers.
Pay-to-Crawl: A New Era for Website Content
However, the dynamic has shifted with AI technology, as consumers are less likely to click through to the source after getting their answers via an AI chatbot.
Therefore, publishers are struggling to recover from the hit AI has had on their bottom line, with search traffic declining significantly.
Moreover, a pay-to-crawl system could help smaller web publishers that don’t have the pull to negotiate one-off content deals with AI providers.
CC’s Caveats and Principles for Responsible Pay-to-Crawl
In addition, CC has suggested a series of principles for responsible pay-to-crawl, including not making pay-to-crawl a default setting for all websites.
Furthermore, CC recommends avoiding blanket rules for the web and allowing for throttling, not just blocking, of AI crawlers.
Therefore, pay-to-crawl systems should preserve public interest access and be open, interoperable, and built with standardized components.
Industry Response and Emerging Standards
Meanwhile, companies like Cloudflare, Microsoft, and startups like ProRata.ai and TollBit are building their own AI marketplaces and pay-to-crawl systems.
Furthermore, the RSL Collective has announced its own spec for a new standard called Really Simple Licensing (RSL), which dictates what parts of a website crawlers can access.
Consequently, Cloudflare, Akamai, and Fastly have adopted RSL, which is backed by Yahoo, Ziff Davis, O’Reilly Media, and others.
Source: Link
