Should You Trust a Chatbot With Your Money? A 10,000-Answer Test Has a Verdict

Mainstream AI models failed 57% of personal finance answers, according to UK fintech firm Saturn. Failure rates climbed to 88% on harder, multi-step queries.

The results came as consumer reliance on chatbots for money questions has grown sharply.

Harder Questions Broke Almost Every Model

The study ran 121 questions through 18 free and paid models from providers including ChatGPT, Gemini, Claude, and Copilot. Each question was repeated up to 5 times, producing more than 10,000 answers.

The test scored an answer as a failure when it contained a factual error, skipped something material, or omitted a required warning.

Free models performed the worst, failing on 63% of answers compared to 49% for paid versions. On the hardest questions, free models failed 93% of the time.

Claude Opus 5 in reasoning mode led the field. However, it still failed 39% of answers. The errors included miscalculations, overlooked tax changes, and nonexistent rules. 

One pension tax answer could have exposed a saver to a £17,500 charge from HM Revenue and Customs.

“Millions of people are trusting the AI models for money advice, but they are getting wrong answers that can lose them money,” Amal Jolly, Saturn chief executive, said.

Follow us on X to get the latest news as it happens

Trust in AI Chatbots Keeps Climbing Anyway

Meanwhile, usage has widened across markets and age groups. A global EY survey of 18,000 consumers found that 49% had used AI to support savings and investment decisions.

Britain’s financial regulator reported in August that four in five less experienced investors have used AI for help with investing. Of those surveyed, 56% said they trust the tools, ahead of television and radio at 47%.

Meanwhile, the same research found 44% wrongly believe AI-generated financial information is regulated.

A PensionBee survey of 1,000 US adults found nearly six in ten would act on money guidance without independently checking it. Nearly one in four said a chatbot had already given them wrong information about their finances.

Jolly said AI financial advice is unregulated, leaving consumers without the compensation rights a human adviser would carry. He urged the FCA to act quickly. 

Subscribe to our YouTube channel to watch leaders and journalists provide expert insights