Family Sues Google Over Teenager’s Suicide-Related AI Chats On School Computer

The parents of a 13-year-old girl have sued Google claiming she planned a potential suicide and funeral using its AI technology on a school-issued computer. Filed Monday in a California court, the complaint is the first known case accusing Google of negligence related to AI integrations on Chromebooks, the tech giant’s low-cost laptops used by millions of students in the United States and around the world.

The teenager, identified by the name Emma, was not able to access Google’s Gemini app, nor OpenAI’s ChatGPT, because they were blocked by her school. She was nonetheless still able to use Google’s search engine, which contains AI-powered tools. Those features were notably not among the tools that Google announced this week will be disabled by default for students.Google on March 21, 2023 invited people in the United States and Britain to test its AI chatbot, known as Bard, as it scrambles to catch up with Microsoft-backed ChatGPT. (Photo by Pau BARRENA / AFP) “Schools should turn search off and keep it off until Google removes AI mode from search or ensures that every single student is always safe when they’re interacting with AI,” Andrew Liddell, one of the family’s attorneys, told AFP on Thursday.

The complaint alleges that Emma had been a “happy 13-year-old girl” in North Carolina who received great grades, played music and participated in Girl Scouts. Despite “no history” of mental illness, within months of using Google’s AI-powered search on her school laptop, she began a “quiet spiral into a severe psychological crisis” according to the complaint.

On March 17, the school’s monitoring software flagged a document titled “Funeral,” and the school notified her parents. Her mother then saw Emma writing a farewell letter in real-time after logging into her daughter’s account, and her father intervened. “We are reviewing the claims in the lawsuit and we take this very seriously,” Google spokesperson Jose Castaneda told AFP.

Google argues that AI mode includes safeguards against self-harm and attempts to bypass them. But Emma “was able to keep going back over and over and over again,” Liddle said. “Whatever safeguards they think they put in place were obviously insufficient.” Developers of fast-evolving AI technology are facing a litany of similar lawsuits in the US. OpenAI faces several wrongful death suits, including from the parents of 16-year-old Adam Raine, a teenager who committed suicide allegedly using methods advised by ChatGPT.

Google and the startup Character.AI agreed in January to settle several lawsuits filed by families of teenagers out of court. On Wednesday, Google, TikTok and Meta each separately sued California to block a law that prohibits social media companies from providing addictive features to users under 16. Governor Gavin Newsom signed the law in September, and it goes into effect on January 1, 2027.

AFP