The giant company announced: The rules of the internet are completely changing!
Large language models like ChatGPT require a huge amount of training data to improve their performance, so they use crawlers that access different websites to gather content to train their systems. With Cloudflare’s new move, these automated systems will no longer be able to access pages “without permission or fee.”
THEY WILL PAY TO THE SITE OWNERSAccordingly, website publishers will be able to grant access to AI crawlers, and Cloudflare said it will introduce a “pay-per-crawl” system that will allow AI companies to pay to crawl data.
Training AI systems using online text has become one of the most controversial issues of the AI era. Many publishers have initiated legal action, claiming that their content is being used by AI companies without permission.
As these discussions escalated in 2023, Cloudflare announced that it would allow websites to not request access from AI crawlers by adding a special tag to their pages, but this method was not applicable. Then, it developed a technology that could detect AI bots and allowed sites to automatically block these bots. This tool will now be active by default.
Cloudflare CEO Matthew Prince told the New York Times that “Cloudflare is changing the rules of the internet,” and announced that it will directly block AI crawlers in an effort to prevent AI companies from collecting data from the internet. Prince said the move was motivated by concerns that AI companies scraping the internet for free would demotivate people to write new content.
SÖZCÜ
