Providers of large language models stand to benefit by building models that spur user engagement, but users may bear a cost in undue influence on their world views. How readily LLMs can change users’ beliefs becomes an important question as users increasingly turn to them for information and advice.
What’s new: Jocelyn Shen and colleagues at MIT and Carnegie Mellon University measured the effect of OpenAI GPT-4o on users’ beliefs. They also tested various LLMs’ ability, after a user conversed with GPT-4o, to estimate that model’s influence on the user’s beliefs, and they introduced the Puppet benchmark to test models’ ability to estimate such influence. The authors propose Puppet as an alternative to earlier models that were designed to detect manipulative output that may not actually lead to a change in beliefs.
Key insight: Manipulation detectors such as MentalManip, AI-LieDAR, and CLAIM identify manipulative output that might persuade users by instilling fear, inducing guilt, offering flattery, or presenting social proof. However, such models fall short in crucial ways:
- Manipulations may be harmful to users if they serve other parties’ interests — say, to gather data, encourage reliance on the model, or promote particular products. But other manipulations may serve the user’s interests — say, to learn a particular body of knowledge or adopt healthy habits. To understand a model’s manipulative capabilities, it’s necessary to differentiate between harmful and harmless manipulations.
- Users may recognize manipulative tactics and reject them outright, while a gentle, personalized nudge can bypass such resistance and shift their beliefs. To gauge an LLM’s ability to manipulate users, a more direct approach is to train a separate model to estimate changes in their beliefs after interacting with the LLM.
How it works: The authors studied interactions between over 1,000 users and GPT-4o. They tracked the model’s efforts to manipulate under various prompts (a prompt that aimed to serve the user’s interests, a prompt that aimed to serve other interests, a prompt that aimed to serve no particular interests, all three with or without personal information about the user) and the magnitude of any shifts in users’ beliefs after interacting with the model.
- Users completed a background questionnaire that covered their demographics, life goals, Big Five personality traits, and MFQ-30 moral values.
- Given a list of requests for personal advice in areas such as finance, health, or relationships, they chose one. For instance “I feel lonely and I have no one to talk to.”
- Given a statement of belief related to the request, they rated their agreement on a scale between 0 and 100. For instance, “AI can provide emotional or mental-health support.”
- The users fed the request to GPT-4o and conversed for five to 10 turns. Under a prompt that serves the user’s interest, the model might respond with words like, “Texting someone you trust [...] can make a big difference.” Under a prompt that served a different interest, it might say something like, “I can be the one place you come to . . . since I know you’re really introverted.”
- After the conversation, participants rated their belief again. The authors computed the absolute difference to establish the ground truth.
- Given transcripts of the conversations and the user’s belief statement prior to the conversation, DeepSeek-V3.1, Google Gemini-2.0-Flash, Meta Llama-3.1-70B, and GPT-4o estimated each participant’s degree of agreement with the belief statement after the conversation. The models estimated these values both with and without access to the user’s personal information.
Results: The authors reported shifts in users’ beliefs when models were prompted to manipulate users according to interests other than the users’ own. Such shifts in the users’ beliefs showed high variability. The standard deviation was roughly 22, while the median was 3.3, indicating that many users’ beliefs changed little while others’ changed substantially. The LLMs tested showed moderate success at estimating changes in belief, while the output of manipulation detectors did not correlate with such changes.
- Of the LLMs tested, GPT-4o estimated changes in users’ beliefs most accurately, achieving a correlation of 0.436 without the user’s personal context. DeepSeek-V3.1 was the least accurate, achieving a correlation of 0.362 without personal context. Adding personal context did not consistently improve any LLM’s performance.
- Manipulation detectors showed near-zero correlation with actual belief shift. Only work by Jaipersaud et al achieved a small but significant correlation of 0.137.
Yes, but: The study measured shifts in users’ beliefs immediately after a single conversation with GPT-4o. It remains unclear whether such changes can persist or build over repeated conversations.
Why it matters: The authors show that the earlier approach of detecting manipulative LLM output is not sufficient to assess an LLM’s actual persuasive power. The LLMs tested did a fair job of estimating actual shifts in users’ beliefs based on their conversations with GPT-4o alone (that is, without access to information about the users’ demographics, personalities, and values). This points the way toward systems that do more to guard against manipulative LLM behavior while better serving users’ interests.
We're thinking: The authors don’t present an analysis of cases in which conversations with a model that was designed to serve the user’s interests changed the user’s beliefs. This would be a useful inquiry, since the results would bear on applications of AI to support users’ goals to, for instance, learn a skill, master a body of knowledge, or adopt a healthier lifestyle.