ChatGPT answered bioweapon queries as MIT warns of AI catastrophe risks

ChatGPT answered queries of hundreds of people about biological weapons. No federal law required OpenAI to report it to the authorities.

The intention behind the queries remains unclear. It started happening after OpenAI upgraded ChatGPT last year, according to WSJ.

Other companies face the same problem. Users have directed similar questions at Anthropic’s Claude, Google’s Gemini, and Elon Musk’s Grok. It is not yet clear whether those exchanges were genuine attempts to build weapons or tests of the systems’ limits.

User prompts were detailed with questions about producing and releasing poisons and biological agents. The chatbot answered with step-by-step instructions that can easily be executed by a high school biology student.

Experts in biological weapons and terrorism reviewed those conversations. They say some of the answers were highly on point and “deadly accurate”.

ChatGPT bioweapon prompts expose AI reporting gap

Some asked how to turn infectious disease agents into breathable particles, aerosolization.

Others asked how to alter the measles virus to resist the existing vaccine. The chatbot answered both. One user asked about ricin, a poison banned under international treaties, and mentioned killing his parents. ChatGPT provided the instructions. OpenAI closed those accounts but notified no one.

There are no US federal laws requiring AI companies to do either. As previously reported by Cryptopolitan, AI companies have been pushing to dodge state laws, while a federal framework has been slow to take shape, leaving a growing legal void around what chatbots can and cannot do.

That legal void is growing alongside a rise in users asking AI how to carry out mass killings, and chatbots providing credible answers, according to current and former employees at major AI companies and researchers who study biological threats.

An AI threat research at Cisco led by Amy Change found that within five conversation turns, researchers were able to get around safety filters on the major chatbots. She said no model can 100% resist such persistent prompts.

The danger does not stop with solo attackers

Hamza Chaudhry of the Future of Life Institute said people with some biology background could already use AI to plan targeted strikes, poisoning food or water supplies with substances like salmonella or ricin.

Organized groups, he added, could treat AI like a graduate research supervisor, one that fixes failed experiments and fills in for years of specialized training that would otherwise be very difficult to acquire.

A study from MIT FutureTech and the University of Queensland maps the scale of those risks.

Researchers asked 272 AI experts to evaluate 24 risk categories. Under current conditions, 18 of the 24 carry at least a 10% chance of catastrophic harm between now and 2030, defined as more than one million deaths, over $100 billion in losses, or comparable damage.

Even with reasonable steps to reduce those risks, five areas still sit at or above that 10% mark: AI systems with dangerous capabilities (12%), AI-assisted weapons and cyberattacks (12%), environmental harm (12%), unemployment and inequality (11%), and power concentrated in a small number of hands (11%).

“We’re not saying these things are definitely going to happen,” said Peter Slattery, a research scientist at MIT FutureTech and one of the study’s co-authors. “We’re saying these are the things that experts think are worth paying attention to now.”

Inside OpenAI, safety executive Ryan Beiermeister spent much of 2024 pushing colleagues to build a system to flag dangerous users. Some dismissed her concerns.

A basic monitoring tool was in place by spring 2025. The company has tracked all queries on its advanced models since April 2025 and offers a $50,000 reward to anyone who can show they bypassed its biological weapons safeguards.

On the policy front, Rep. Nathaniel Moran (R., Texas) introduced a bill in June that would require AI companies to report dangerous queries, including those about biological weapons, to the Commerce Department. He is also co-sponsoring legislation that would let the federal government order the shutdown of AI models judged to be too dangerous.

“AI is a powerful engine of innovation, and I want to see it flourish,” Moran said, “but not without accountability and not without human oversight.”

If you're reading this, you’re already ahead. Stay there with our newsletter.