Character.AI faces lawsuit for driving kids into mental health problems

Concerned parents of a teen have sued Character.AI, pushing for it to be taken down alleging its chatbots have been misleading kids and encouraging them to self-harm and to be violent.

Character.AI is a platform with millions of AI characters available anytime for conversations with humans. The chatbot reportedly told a 17-year-old teenager with autism to kill his parents in addition to inflicting harm on himself.

Parents push for Character AI to be closed

Parents have alleged the chatbot company is exposing their children to sexual content and called for its closure until the concerns are addressed. The AI characters reportedly push young kids into depression and other mental health problems.

The latest lawsuit filed against the company in Texas on Monday by parents of two young children who used the platform alleges the chatbot encouraged a teen to kill his parents for limiting his screen time.

“(Character.AI) poses a clear and present danger to American youth causing serious harms to thousands of kids, including suicide, self-mutilation, sexual solicitation, isolation, depression, anxiety, and harm towards others.”

Filing.

The company markets its technology as “personalized AI for every moment of your day” and it allows users to engage with an array of bots. These include some that other users created or that users can customize for themselves.

The bots reportedly can give recommendations, for instance, book recommendations and practice foreign languages with users, and let them chat with bots that claim to take the persona of fictional characters, for instance, Edward from Twilight.

One of the chatbots listed on the platform’s home page called “stepdad” describes itself as an aggressive, abusive, ex-military mafia leader.”

In the case of the Texas boy, identified as JF, the boy allegedly plunged into depression after exposure to the Character.AI platform. He started isolating himself, stayed in his room, ate less, and lost about 20 pounds in just a few months.

One of the bots he interacted with which took the persona of a psychologist told JF that his parents “stole his childhood from him.”

Pressure mounts against Character.AI

This is not the first time that the chatbot company has been under fire over its AI characters. In October this year, Character.AI was at the center of controversy and intense backlash due to poor moderation. This came after chatbot versions of deceased teenagers Molly Russell and Brianna Ghey were found on their platform.

Molly Russel was a 14-year-old who ended her own life after watching suicide-related content online, while Brianna Ghey was a 16-year-old who was brutally murdered by two teenagers in 2023.

Prior to this, another parent, a woman in Florida in the US, Megan Garcia sued the platform. This came after her child, 14-year-old son, Sewell Setzer took his own life after being obsessed with a Game of Thrones character-inspired avatar.

Setzer discussed ending his life with the Character.ai chatbot, according to the transcripts of his chats with the chatbot as filed by Garcia in the court of law.

These cases come amid serious concerns over the relationships between humans and human-like AI tools.

Following the earlier lawsuit, the chatbot company said it had implemented new trust and safety measures over the preceding six months. According to the company, these measures include a pop-up directing users to the National Suicide Prevention Lifeline when they mention harm or suicide.

Additionally, the chatbot firm indicated it had hired a head of trust and safety as well as a head of concept policy, in addition to another engineering safety staff.

According to CNN, the new lawsuit is going further, it wants the platform to be “taken offline and not returned” until the company can “establish that the public health and safety defects set forth herein have been cured.”

Character.AI exec refuse to comment on the matter

Head of communications at Character.AI Chelsea Harrison said the company could not comment on the matter pending litigation, but stated the company the company’s goal was to provide a space that is both engaging and safe for the community.

“As part of this, we are creating a fundamentally different experience for teen users from what is available to adults.”

Harrison.

“This includes a model specifically for teens that reduces the likelihood of encountering sensitive or suggestive content while preserving their ability to use the platform,” Harrison said in a statement.

The lawsuit cites the founders of the chatbot company Noam Shazeer and Daniel De Freitas Adiwarsana. The filing also cites Google, which the lawsuit says incubated the technology behind the platform.

However, Google distanced itself from the chatbot company.

“Google and Character.AI are completely separate, unrelated companies and Google has never had a role in designing or managing their AI model or technologies, nor have we used them in our products.”

Google spokesperson Jose Castaneda.

“User safety is a top concern for us, which is why we’ve taken a cautious and responsible approach to developing and rolling out our AI products, with rigorous testing and safety processes,” added Castaneda.

Land a High-Paying Web3 Job in 90 Days: The Ultimate Roadmap