Major publisher accuses Google of not playing fair with AI web crawler tactic

People CEO Neil Vogel has criticized web crawlers, accusing Google, the technology heavyweight, of being a bad actor. According to reports, the CEO of the publishing firm that operates over 40 brands accused Google of crawling websites to support its AI products.

People, formerly known as Dotdash Meredith, is a publisher that operates brands like Wine, People, Travel and Leisure, Southern Living, Better Homes and Gardens, and People.

According to Vogel, Google is not playing fair because it uses the same bot to crawl websites to index them for its search engine and support its AI features. “Google has one crawler, which means they use the same crawler for their search, where they still send us traffic, as they do for their AI products, where they steal our content,” said Vogel.

People CEO goes after Google for being a bad actor

According to Vogel, Google Search represented more than 65% of the company’s traffic, and the number has since dropped to a range close to the high 20s. He also shared another statistic that was made available by AdExchanger last month, which noted that as of several years ago, Google’s traffic accounted for about 90% of People Inc.’s traffic from the open web.

“I’m not complaining. We’ve grown our audience. We’ve grown our revenue,” Vogel told conference attendees. “We’re doing great. What is not right about this is: You cannot take our content to compete with us.”

Vogel mentioned that he believes publishers need to be given more in the AI era, which is one of the reasons why he feels it is important to block AI crawlers–automated programs that scan websites to train AI systems–as it forces them into content deals.

Vogel mentioned that his company presently has a deal with OpenAI, which he describes as a good actor in the space. People say it has been using web infrastructure company Cloudflare’s latest solution to block AI crawlers that do not pay, with the move forcing AI first to approach the publisher with potential content deals.

However, People cannot afford to block Google’s crawler, according to Vogel. “They know this, and they’re not splitting their crawler. So they are intentionally bad actors here,” Vogel declared.

Experts wade into the discussion surrounding AI crawlers

The same view was shared by Janie Min, the editor-in-chief and CEO of Ankler Media, calling Big Tech, like Google and Meta, content kleptomaniacs. “I don’t see the benefit to us in partnering with any AI company right now,” she said, adding that her company blocks AI crawlers.

Meanwhile, Cloudflare CEO Matthew Prince, whose company makes the AI-blocking solution and was also on the panel, says he believes that things could still change in the future when it comes to how AI firms behave. He added that these changes could be introduced as part of the new regulations. The Cloudflare executive also questioned the need to fight AI companies using legal means, asking if it was the right answer.

“I think that it’s a fool’s errand to go down that path, because, in copyright law, typically, the more derivative something is, the more it’s protected under fair use … What these AI companies are doing is they’re actually creating derivatives,” Prince said. “And so if you look at the best case law that’s come out so far, it’s actually said that the use by Anthropic and others — the reason Anthropic settled recently with all the book publishers for $1.5 billion — was for them to be able to preserve the positive copyright ruling that they got.”

Don’t just read crypto news. Understand it. Subscribe to our newsletter. It's free.