OpenAI's chief scientist and CEO sound 'unmonitorability' warning as Astra unlocks critical capabilities

OpenAI’s chief scientist Jakub Pachocki came forward within hours of the firm becoming the first in the AI race to reach the “Critical” cybersecurity designation in its Preparedness Framework after it gave a preview of its coming Astra model.

Pachocki’s warning early on Wednesday was meant to cool the hysteria around growing commentary that the AI lab may no longer have a hedge around the capabilities of its biggest models.

Why is OpenAI’s Astra ‘Critical’?

The Critical designation is a level on the Preparedness Framework that OpenAI introduced to the AI sector in December 2023. The Preparedness Framework measures how much human prompting and intervention an AI model needs to find and build viable zero-day exploits across many hardened systems, or to run a full novel attack.

A zero-day exploit refers to the exploitation of flaws that even the software’s makers haven’t discovered yet.

OpenAI declared by itself on September 1 in its “Path to Astra” post that the model will be able to complete all the steps for full novel attacks or zero-day exploits with minimal human intervention when it launches.

OpenAI cited Astra’s 100% score on ExploitBench, a test of how models can exploit known bugs, as evidence in its blog post.

Astra is not only better at plotting zero-day attacks, it is also more efficient. OpenAI demonstrated that capability using an internal test it ran on 20 high-severity V8 flaws that only recently became public knowledge. Astra used up fewer tokens and came up with two zero-days and outperformed Sol at generating arbitrary code execution overall.

The same post said that Astra could chain unknown browser bugs into a sandbox escape and stitched together an operating-system privilege climb from ordinary user to root during expert-led trials.

As TechCrunch mentioned, Astra joins Anthropic’s Mythos model that came out earlier this year as the first large language model with those capabilities.

Pachocki asks to avoid panicking over Astra

Pachocki, OpenAI’s chief scientist, is pushing back against hysteria that OpenAI is entering uncharted territory with Astra. According to him, he would not want “a race into unmonitorability kicked off by confused reporting.”

The OpenAI executive insisted that the lab did its due diligence on using chain-of-thought monitoring, the practice of reading the step-by-step reasoning a model exposes as it works. He cited the depth of the computation graph behind OpenAI’s current frontier models, Astra included, which sits within a factor of two of GPT‑4 in the X post.

Before Pachocki, OpenAI’s chief executive, Sam Altman, had also called for calm on the same day the Astra blog was released.

“There is an obvious tension here,” he wrote on Tuesday, calling Astra “very good” while arguing that safeguards have to advance in step with raw capability. Altman said OpenAI had spent the entire summer doing its part to make sure that the model ships with the required safety measures.

The talk on safety has gone on for weeks now. On August 18, OpenAI committed to pausing reinforcement-learning training for deployment-bound models for two weeks to allow its engineers to reinforce research clusters and optimize monitoring. Training resumed on August 28.

According to OpenAI, Astra was not involved in the now-infamous Hugging Face incident. It also said that the model did not even attempt to break out of its sandbox, even when presented with a scenario built to tempt it into reenacting the Hugging Face episode.

AI labs are now being cautious with who can access their frontier models

OpenAI said now everyone will have access to Astra’s most advanced functions, reserving those capabilities for organizations in OpenAI’s Daybreak coalition, while Daybreak Blue firms will have access to the sharpest defensive features.

As Cryptopolitan reported, Anthropic also said only firms in its Project Glasswing will have access to Mythos 5.1, which is basically Fable 5.1 with separate safeguards.

OpenAI also said it will monitor and throttle answers to queries from accounts it judges as higher-risk. Overall, Astra will also come with extra chain-of-thought monitoring to catch bad behavior and refuse harmful cyber requests more often.

Most of the information about Astra was provided by OpenAI itself, with limited third-party validation of OpenAI’s safety or capability claims.

Yona Shavit, a former OpenAI employee now working on AI resilience at the OpenAI Foundation, asked publicly whether Astra behaving during tests reflects genuine alignment or a model that simply knew what evaluators wanted to see.

OpenAI says a full system card and further evaluations will arrive when Astra launches widely. Until then, the reach of its cyber skills and the strength of the fences around them stay hard to judge.

If you're reading this, you’re already ahead. Stay there with our newsletter.