Dario Amodei's AI Safety Warning Is Colliding With Anthropic's IPO Timeline

Anthropic CEO Dario Amodei has warned repeatedly in recent months that artificial intelligence (AI) models could advance faster than the safeguards needed to control them. Yet Anthropic is still reportedly aiming to conduct its initial public offering (IPO) in November.

On Oct. 14, New York City Comptroller Mark Levine urged the company to delay its IPO until it addresses concerns about AI safety and governance. According to Bloomberg, Anthropic subsequently contacted Levine, but no change to the IPO timeline was reported.

Anthropic CEO Dario Amodei. Image source: Getty Images.

Safety concerns have already interrupted AI development

In an essay published in September, Amodei expressed concern that AI's "recursive self-improvement" -- the systems' ability to change their own programming and self-evolve -- could lead to the development of powerful AIs that go beyond their developers' ability to keep them in check or understand their behavior. Existing safeguards, in his view, will not be enough to keep pace.

He has called for independent safety reviews and greater cooperation among AI companies and governments. Shane Legg, co-founder of Alphabet-owned Google DeepMind, has also warned that AI advances are at risk of outpacing safety controls.

These concerns have already affected Anthropic's operations. On Aug. 31, Anthropic reported it had paused some high-risk training activities involving unreleased AI models for several weeks. Most of those pauses affected reinforcement-learning work that resumed after Anthropic introduced additional monitoring.

On Oct. 9, Anthropic reported additional misalignment and security incidents related to its AI models. The company has also announced plans to extend its restrictions on live internet access to all internal model evaluations until its monitoring systems can reliably detect similar incidents. The financial impact these restrictions will have on the company remains unclear.

Anthropic's AI safety decisions

Anthropic is taking steps to strengthen independent safety reviews, but outside evaluators cannot stop the company from releasing an AI model. On Oct. 15, Anthropic announced that six researchers from the nonprofit METR had received access comparable to its internal safety teams. METR can publish its findings without Anthropic's approval, but it does not have any authority to decide whether or not AI models should be released.

Anthropic's Long-Term Benefit Trust appoints a majority of its board members to help balance financial interests with AI safety. Reuters has also reported that the company's draft IPO structure would give its seven co-founders 50.1% of voting power on key corporate matters through a new entity called Founder LLC. Additionally, Anthropic's draft prospectus warned that this governance arrangement could result in decisions that run counter to those that would put financial performance or shareholder value first. Hence, if safety-related delays affect Anthropic's financial performance, public shareholders will bear the consequences, but they will have limited influence over major corporate decisions.

Anthropic is also seeking to get the U.S. government more directly involved in AI safety restrictions. According to Reuters, the company has been pitching draft legislation to Congress that would allow the Department of Commerce and federal courts to block certain AI models if they were deemed dangerous. If enacted, the law could delay the commercialization of AI models and potentially affect the growth expectations that underpin Anthropic's planned IPO valuation.

Safety decisions could affect Anthropic's IPO valuation

Anthropic's revenue rose 45% sequentially to $16.8 billion in the third quarter. However, some key investors invited to Anthropic's investor day in San Francisco last week were reportedly discussing whether it would be more appropriate for it to IPO at a valuation of around $1.5 trillion, down from its target of $2 trillion. These investors are concerned about the slowing revenue growth rate and rising AI infrastructure costs. There is no established evidence that safety-related restrictions contributed to this valuation reassessment.

Hence, while Amodei's warnings about the dangers of AI have not derailed Anthropic's IPO timeline, the company will still need to demonstrate that it can implement the necessary safety measures while simultaneously delivering the growth and profits needed to justify its ambitious valuation.

Manali Pradhan, CFA has no position in any of the stocks mentioned. The Motley Fool has positions in and recommends Alphabet. The Motley Fool has a disclosure policy.