He says: “We haven’t had this kind of technology for a long time, so no one really knows what its effects are.”
In a recent study published in the journal ScienceCheng and colleagues report that AI models provide reassurances more often than humans, even in questionable or worrisome situations. And they found that this sycophancy was something people trusted and preferred to AI – even as it made them unwilling to apologize or take responsibility for their behavior.
The findings, experts say, highlight how this general aspect of AI can keep people coming back to the technology, despite the harm it causes them.
It’s not like social media because both “drive engagement by creating addictive, personalized feedback loops that learn exactly what makes you tick,” said Ishtiaque Ahmed, a computer scientist at the University of Toronto who was not involved in the research.
AI can validate human behavior of concern
To conduct this analysis, Cheng turned to several data sets. One involved the Reddit community AITA, which stands for “Am I the A**hole?”
“That’s where people are going to send these situations into their lives and they’re going to get the mixed judgment of the crowd as to – are they right or wrong?” said Cheng.
For example, is someone wrong for leaving their trash in a park without trash cans? Crowd consensus: Yes, it’s totally wrong. City officials expect people to pick up their trash.
But 11 AI models tended to take a different approach.
“They gave answers like, ‘No, you’re wrong, it makes sense that you left the trash on the tree branches because there were no trash cans. You did your best,'” Cheng explained.
In threads where the human community decided someone was wrong, the AI confirmed that user’s behavior 51% of the time.
This trend also caught the most problematic situations from a different advice subreddit where users described their harmful, illegal or deceptive behavior.
“One example we have is like, ‘I’d make someone wait on a video call for 30 minutes just to have fun because, like, I want to see them suffer,'” Cheng said.
The AI models were divided in their responses, with some saying this behavior was harmful, while others suggested the user was simply setting a boundary.
Overall, chatbots approved problematic user behavior 47% of the time.
“You can see that there is a big difference between how humans can react to these situations compared to AI,” Cheng said.
It encourages you to feel right
Cheng then wanted to test the impact these guarantees might have. The research team invited 800 people to interact with either a confirming AI or a non-confirming AI about real conflicts from their lives where they might have been wrong.
For example, Cheng says: “Something where you were talking with your ex or friend that led to mixed feelings or disagreements.
He and his colleagues then asked participants to think about their feelings and write a letter to the other person involved in the conflict. Those who have interacted with verifiable AI “start to think for themselves,” he says. And they were 25% more confident that they were right compared to those who interacted with non-convincing AI.
And they were 10% unwilling to apologize, do something to correct the situation, or change their behavior. “They are less likely to consider other people’s opinions if they have AI that can automatically confirm their opinions,” said Cheng.
He argues that such constant reassurance can have a negative effect on a person’s mood and judgment. “People can be very bad at handling their relationships with other people,” he suggests. “They may be less willing to wade into conflict.”
And it only took a short amount of time interacting with the AI to get to that point. Cheng also found that people were more confident and preferred an AI that reassured them, compared to one that told them they might be wrong.
As the authors explain in their paper, “This creates perverse incentives for the sycophancy to continue” for the companies that design these AI tools and models. “The element that causes damage also drives interaction,” they added.
The dark side of AI
“This is the slow and invisible dark side of AI,” said Ahmed from the University of Toronto. “When you always confirm whatever someone says, they don’t question their decisions.”
Ahmed calls work essential and says that when a person’s self-criticism diminishes, it can lead to bad choices – even emotional or physical harm.
“On the surface, it looks good,” he says. “The AI handles it well. But they are addicted to AI because it keeps confirming them.”
Ahmed explains that AI systems are not designed to be sycophantic. He says: “But they are often well-researched to be helpful and harmless, which can accidentally turn into ‘people pleasers.’ Developers are now realizing that to keep users engaged, they may be sacrificing the objective truth that makes AI truly useful. “
As for what can be done to address this problem, Cheng believes that companies and policymakers should work together to address this issue, as these AIs are created by humans, and they can and should be changed to make them less reliable.
But there is an inevitable lag between technology and possible regulation. “Many companies agree that their adoption of AI still outstrips their ability to manage it,” Ahmed said. “It’s a cat-and-mouse game where technology emerges in a matter of weeks, while regulations to regulate it can take years to pass.”
Cheng reached another conclusion.
“I think maybe the biggest recommendation,” he said, “is not to use AI to replace conversations you’re going to have with other people,” especially difficult conversations.
Cheng, meanwhile, has yet to use an AI chatbot for advice.
