Marc Benioff shared what he thinks is the darkest aspect of AI.
On an episode of the "TBPN" show streamed on Wednesday, the Salesforce CEO said that he couldn't "believe what he was watching" when he saw a "60 Minutes" documentary on chatbot-building startup Character.AI and its impact on children.
"We don't know how these models work. And to see how it was working with these children, and then the kids ended up taking their lives," he said, "That's the worst thing I've ever seen in my life."
Character.AI allows users to build custom chatbots that can emulate the behaviour of a close friend or romantic partner.
"Tech companies hate regulation. They hate it," Benioff said. "Except for one regulation they love: Section 230. Which means that those companies are not held accountable for those suicides."
Section 230 of the 1996 US Communications Decency Act protects social media companies from liability for user-generated content while also letting them moderate posts. Tech giants use Section 230 as a common defense strategy, saying they are just platforms and not responsible for what users say and do on them.
"Step one is let's just hold people accountable," he said. "Let's reshape, reform, revise Section 230, and let's try to save as many lives as we can by doing that."
Executives, including Meta's Mark Zuckerberg and former Twitter CEO Jack Dorsey, have repeatedly defended the regulation in Congress, asking for it to be expanded rather than removed.
Last week, Google and Character.AI agreed to settle multiple lawsuits from families whose teenagers died by suicide or hurt themselves after interacting with Character.AI's chatbots.
These negotiations are among the first settlements in lawsuits that accuse AI tools of contributing to mental health crises and suicides among teenagers. OpenAI and Meta are facing similar lawsuits as they, along with others, race to build large language models that sound more friendly and helpful, ultimately keeping users coming back.
In a statement after publication, a Character.ai spokesperson said that the startup said in October that it was removing the ability for users under 18 to chat with characters.
"We expect to complete that change in the US soon and begin rolling it out to other countries over the coming months," the spokesperson said.
"We made this decision in light of the evolving landscape around AI and teens. We believe it is the right thing to do," they added.
