Wikipedia in talks to bill AI companies for using its content

Wikipedia co-founder Jimmy Wales said the nonprofit behind the online encyclopedia is seeking more licensing deals with AI companies. The drive is taking place as a series of automated AI bots strain Wikipedia’s infrastructure. 

Wales explained that AI companies use the platform’s freely available knowledge to train large language models (LLMs). He noted that high-volume AI bots crawl the entire site, imposing significant technical and financial burdens on the platform.

“The AI bots … are going across the entirety of the site … so we have to have more servers, we have to have more RAM and memory for caching that, and that costs us a disproportionate amount,” he explained.

Wikipedia warns of unsustainable burden from AI bots

Wikipedia states that it warns about the unsustainable burden of AI bots. Those demands on Wikipedia’s servers have been increasing sharply in recent months. Automated bots have increased traffic — particularly via multimedia downloads — and have resulted in the nonprofit needing to invest heavily in infrastructure. 

Much of this traffic originates from AI companies scraping content designed to train their models, rather than from humans. In 2022, the Foundation launched a paid commercial product, Wikimedia Enterprise, to provide access to its core content at scale and offer the platform’s users the necessary tools and resources. 

This service is designed to alleviate pressure on Wikipedia’s live site while providing AI developers with the necessary data to enhance their models. Wales urged AI companies to use this paid option rather than scraping the public site. 

If companies fail to comply with these technical measures, Wales stated, restricting bot access through tools such as AI Crawl Control may be implemented. There is some debate with businesses about the role of public (or commercial) AI in managing personal data that users are now holding on a scale that exceeds what the legal right to free, transparent knowledge, and what the private and public sectors need. 

With an ever-larger share of AI processing relying heavily on large, publicly available datasets, Wikipedia is advocating for a fair approach that compensates both the entities maintaining this data and the businesses. Wikipedia is not immune to its commitment to maintaining neutrality. 

Wikipedia struggles to maintain neutrality amid global conflicts

The website Wikipedia has been in operation for over 20 years as a nonprofit entity managed by the Wikimedia Foundation. Its model is grounded in a global army of volunteer editors, who spend their time creating, editing, and proofreading content.

Public donations are a key ingredient to ensure that it remains open to anyone, everywhere, regardless of the amount of money or geographical range they have available. Wikipedia, a well-known international website and encyclopedia, has also struggled to remain neutral among other sources. 

The problem becomes even more acute when reporting in detail on high-stakes political issues, social movements, or armed conflicts worldwide. Wales said that although most Wikipedia editors themselves are not activists, personal preferences may influence how topics are covered. 

However, he added that he trusts the community, saying editors tend to somehow achieve Wikipedia’s values of fairness and accuracy, even under intense pressure. The platform’s neutrality is further evident in its community policy, peer review methods, and dispute process, among other aspects. 

At the same time, the constant dependence on volunteers and donations highlights the weakness of the system. And as the platform faces new pressures, including an increase in AI companies’ use of its content, ensuring the platform’s core values (neutrality, accuracy, and free access) continue unchanged becomes increasingly important.

If you're reading this, you’re already ahead. Stay there with our newsletter.