OpenAI’s o3 model falls short of its own benchmark claims

Source Cryptopolitan

OpenAI’s newest LLM, o3, is facing scrutiny after independent tests found it solved a far fewer number of tough math problems than the company first claimed. 

When OpenAI unveiled o3 in December, executives said the model could answer “just over a fourth” of the problems in FrontierMath, a notoriously hard set of graduate‑level math puzzles.

The best competitor, they added, was stuck near 2%. “Today, all offerings out there have less than 2%,” Chief Research Officer Mark Chen said during o3 and o3 mini livestream. “We’re seeing, with o3 in aggressive test‑time compute settings, we’re able to get over 25%.”

TechCrunch reported that the result was obtained by OpenAI on a version of o3 that used more computing power than the model the company released last week.

On Friday, the research institute Epoch AI, which created FrontierMath, published its own score for the public o3.

Using an updated 290‑question edition of the benchmark, Epoch put the model at about 10%.

The result does match with a lower‑bound figure in OpenAI’s December technical paper, and Epoch cautioned that the discrepancy could be due to various reasons.

“The difference between our results and OpenAI’s might be due to OpenAI evaluating with a more powerful internal scaffold, using more test‑time computing, or because those results were run on a different subset of FrontierMath,” Epoch wrote.

FrontierMath is designed to measure progress toward advanced mathematical reasoning. The December 2024 public set contained 180 problems, while the February 2025 private update expanded the pool to 290.

Shifts in the question list and the amount of computing power allowed at test time can cause large swings in reported percentages.

OpenAI confirmed the public o3 model uses less compute than the demo version

Evidence that the commercial o3 is lacking also came from tests by the ARC Prize Foundation, which tried an earlier, larger build. The public release “is a different model… tuned for chat/product use,” ARC Price Foundation posted on X, adding that “all released o3 compute tiers are smaller than the version we benchmarked.”

OpenAI employee Wenda Zhou offered a similar explanation during a livestream last week. The production system, he said, was “more optimized for real‑world use cases” and speed. “We’ve done [optimizations] to make the model more cost efficient [and] more useful in general,” Zhou said, while acknowledging possible benchmark “disparities.”

Two smaller models from the company, o3‑mini‑high and the newly announced o4‑mini, already beat o3 on FrontierMath, and OpenAI says a better o3‑pro variant will arrive in the coming weeks.

Still, it shows how benchmark headlines can be misleading. In January, Epoch was criticized for delaying disclosure of OpenAI funding until after o3’s debut. More recently, Elon Musk’s startup xAI was accused of presenting charts that overstated the capabilities of its Grok 3 model.

Industry watchers say such benchmark controversies are becoming an occurrence in the AI industry as companies  race to capture headlines with new models.

Cryptopolitan Academy: Tired of market swings? Learn how DeFi can help you build steady passive income. Register Now

Disclaimer: For information purposes only. Past performance is not indicative of future results.
placeholder
AUD/USD: Current price action is likely the early stages of a recovery – UOB GroupAustralian Dollar (AUD) is likely to trade in a sideways range between 0.6220 and 0.6290. In the longer run, current price action is likely the early stages of a recovery phase that could potentially reach 0.6350, UOB Group’s FX analysts Quek Ser Leang and Lee Sue Ann note.
Author  FXStreet
Jan 22, Wed
Australian Dollar (AUD) is likely to trade in a sideways range between 0.6220 and 0.6290. In the longer run, current price action is likely the early stages of a recovery phase that could potentially reach 0.6350, UOB Group’s FX analysts Quek Ser Leang and Lee Sue Ann note.
placeholder
Five bullish Shiba Inu (SHIB) Price Predictions for April 2025SHIB price targets diverge as investors weigh Shibarium L3 upgrades, burn-rate surges, and altcoin market sentiment. Forecasts range from a conservative $0.000012 to a parabolic $0.00030.
Author  FXStreet
Apr 16, Wed
SHIB price targets diverge as investors weigh Shibarium L3 upgrades, burn-rate surges, and altcoin market sentiment. Forecasts range from a conservative $0.000012 to a parabolic $0.00030.
placeholder
Ethereum Price Stays Resilient — Upside Break May Be AheadEthereum price started a downside correction below the $1,780 level. ETH is now consolidating near the $1,800 zone and might aim for a move above $1,820.
Author  NewsBTC
Yesterday 03: 52
Ethereum price started a downside correction below the $1,780 level. ETH is now consolidating near the $1,800 zone and might aim for a move above $1,820.
placeholder
Gold price slides back closer to $3,300 amid tariff deals optimismGold price (XAU/USD) struggles to capitalize on the previous day's bounce from the vicinity of the $3,265-3,260 pivotal support and attracts fresh sellers during the Asian session on Tuesday.
Author  FXStreet
20 hours ago
Gold price (XAU/USD) struggles to capitalize on the previous day's bounce from the vicinity of the $3,265-3,260 pivotal support and attracts fresh sellers during the Asian session on Tuesday.
placeholder
EUR/USD ticks lower despite uncertainty over US-China tradeEUR/USD edges lower to near 1.1400 during European trading hours on Tuesday. The major currency pair ticks lower as the US Dollar (USD) steadies, but remains broadly on edge amid escalating uncertainty about the trade outlook between the United States (US) and China.
Author  FXStreet
18 hours ago
EUR/USD edges lower to near 1.1400 during European trading hours on Tuesday. The major currency pair ticks lower as the US Dollar (USD) steadies, but remains broadly on edge amid escalating uncertainty about the trade outlook between the United States (US) and China.
goTop
quote