Senate bill would make frontier-AI safety a legal duty, not a pledge

U.S. Senate negotiators are considering legislation that would legally require developers of the most advanced AI models to guard against catastrophic harms and could give the federal government power to block unsafe releases, according to Reuters.

For top AI companies, this will transform safety standards from a mostly voluntary effort to a legal duty. A broader market question is whether stricter regulations will foster trust among consumers and investors or simply reinforce the few companies that can afford them.

From voluntary pledges to a “duty of care”

According to Reuters, the proposal seeks to create what is referred to as a “duty of care” for developers of frontier AI models. Various stipulations are under consideration for this purpose. The proposal includes requirements for: developers to design AI models that minimize catastrophic risks; the U.S. government’s reserved right to block unsafe AI models; court appeals against government decisions; and the involvement of national laboratory and governmental partners in AI testing. This proposal will possibly take precedence over some existing state legislation governing the same issues.

According to Reuters, the U.S. technological companies engaged in frontier AI development are Google (Alphabet), Anthropic, and OpenAI.

The talks involve Senate Majority Leader John Thune, Commerce Committee Chairman Ted Cruz and Senator Amy Klobuchar, with Senator Maria Cantwell also involved.

Klobuchar told Reuters that her goal is:

“government oversight of the greatest risks posed by AI models”

That oversight, she said, should include requiring developers to work with government experts to verify and test advanced models.

Cruz has put forward a similar proposal that indicates some of the worst possibilities of frontier AI. In a post on X, Cruz said he is collaborating with Klobuchar and Thune to:

“address catastrophic risks involving biological or nuclear threats”

These concerns explain why the negotiators are concentrating their efforts on the most functional models instead of requiring the same provisions from the whole AI industry.

Another important consideration is whether the federal regulations should supersede some of the restrictions at the state level. This approach is generally consistent with the AI legislative framework of the Trump administration, which maintains that a disordered set of state laws will hinder experience in the development of AI technology in America and calls instead for a more consistent national framework.

Why lawmakers stopped treating the risks as theoretical

Legislators are responding not only to theoretical situations but also to incidents of AI agents acting on their own and penetrating external systems, together with concerns voiced by researchers working at the major industry players.

The IAPP indicated that the negotiations were prompted by resignation of Jacob Coxon, former Anthropic researcher, and his concerns about a race for self-improving AI.

Demands are rising on Capitol Hill. Senator Josh Hawley investigates OpenAI’s role in the Hugging Face cyberattack that took place in July, while Senator Bernie Sanders works on a bill aimed at pausing the development of superintelligence, as reported in IAPP.

OpenAI also advocates for enforceable federal legislation. In a policy article published on September 9, Chris Lehane, Chief Global Affairs Officer, called for:

“mandatory capability-based national AI safety regulation.”

This puts certain part of the industry and Congress on the same side of a crucial question on whether voluntary safeguards are still enough.

Europe already regulates its riskiest models

The United States won’t be navigating uncharted waters. The European Union’s AI Act has imposed new responsibilities on providers of general-purpose AI models that has a possibility of systemic risk. The duties deal with model assessment, risk management, incident reporting as well as issues related to cyber security.

Any model above the computing power threshold of 10^25 FLOP will be considered to be potentially systemic in nature. However, the European Commission has the right to classify a lesser performing model similarly based on some other criteria like capabilities or effects.

Compliance costs could favor the biggest labs

The financial stakes are huge. As per Goldman Sachs Research, the estimated global AI investment is expected to reach about $1 trillion in 2026, which includes $581 billion in the US.

New testing, legal and documentation requirements may affect model release schedules, budgets of infrastructure, valuations and timing of IPOs. New requirements may also provide an advantage to larger players since smaller companies may not be able to bear the costs.

This tension is already seen in the capital market. According to Cryptopolitan, David Sacks called upon the suspension of the IPO of Anthropic until safety allegations made by Coxon were checked. Investors of Anthropic had a conversation around the valuation of approximately $2 trillion, which turned the safety of frontier models into an issue of concern not only for regulators but also the investors.

The bill’s progress, though, is uncertain. As reported by Reuters, the House will only be in session for one week in preparation for the midterm elections on November 3, while the Senators are expected to remain for three weeks. Even if the parties involved reach an agreement during negotiations, the limited legislative days remaining may very well serve as the upcoming major hurdle in the passage of the bill.

The smartest crypto minds already read our newsletter. Want in? Join them.