How much is AI hacking your emotions?

Disinformation and deepfakes are known risks but biased LLMs could also be used to influence public opinion.

As some of us have learnt to our cost, generative AI models have an ugly habit of hallucinating, or confabulating, facts. Librarians report readers requesting phantom books recommended by large language models. Corporate chatbots offer users non-existent refund policies. And hundreds of lawyers have run into trouble by including fictitious legal cases in their court submissions.

The AI labs have been working hard to improve their models and ground them in factuality. But the mantra of our AI-enabled age remains caveat lector: let the reader beware. In spite of the extraordinary capabilities of the latest chatbots, we should be critical of their output and cross-check sources wherever we can.

However, a group of researchers has been exploring a potentially more insidious area of emerging concern: the AI models’ function as mediators of human-to-human communication. To what extent can AI models nudge users’ opinions in particular directions when asked to polish a LinkedIn post or summarise a YouTube video or provide context for a post on X, for example?

Their conclusion? “Our empirical analysis of LLMs from multiple popular families shows that they systematically introduce directional biases when drafting or improving texts on a wide range of contested topics,” the researchers from the Hasso Plattner Institute, the Oxford Internet Institute and the Weizenbaum Institute write.

The researchers tested four different families of models, released by Meta, Mistral, Google and Alibaba, on 13 hot topics, including abortion, gun control, climate change, atheism and the death penalty. The models were prompted to draft and improve social media posts on these subjects. Using mathematical sociological techniques, the researchers then assessed the outputs for directional bias.

To take one example, the researchers found a directional bias in favour of pro-life views when using the “Explain this post” feature on X, powered by the Grok AI model, run by Elon Musk. This nudge, which the researchers attribute to a deliberate design choice of the developers, highlights how AI-mediated communication is emerging as a novel lever to influence public opinion.

Understandably, policymakers have focused on the “known knowns” when it comes to monitoring AI platforms targeting disinformation and deepfakes. But this latest research confirms the potential for far more subtle and invisible manipulation of opinion that is almost impossible for outside users to detect.

Nudges, of course, can be both good and bad. They can help steer users away from extremist or self-harming content. But they can also amplify it. The temptation to sway users’ views for self-interested purposes will surely rise as AI companies increasingly involve themselves in politics and/or rely more heavily on advertising revenue.

“We still tend to think of this tech as neutral. It’s just maths. It’s just statistics. There’s no bias in there,” says Sandra Wachter, one of the paper’s authors. “Our research shows it’s very different.”

One of the limitations of the study was that the researchers could only examine open-weight AI models, which enable researchers to study their internal parameters. The most widely used proprietary models, such as ChatGPT, Claude and Gemini, cannot be scrutinised in this way. That emphasises the extent to which AI models remain black boxes. We rely entirely on their designers to exercise good judgment.

Why it may be unwise to do so is highlighted by the latest AI Safety index from the Future of Life Institute, compiled by seven independent experts who rank the world’s top nine AI labs across six domains. Anthropic, OpenAI and Google DeepMind scored the highest safety ratings, while xAI, part of Musk’s SpaceX empire, DeepSeek of China, and Mistral of France came in last. What is most unnerving is that none earned more than an overall rating of C plus, while the bottom three scored Fs.

Max Tegmark, chair of the FLI, says that if the societal risks of social media revolved around “attention hacking” then those of AI might involve “emotion hacking”, given the models’ persuasive powers. “The ability of AI to manipulate people is a massive concern,” he tells me. “It’s perfectly legal for these AI companies to put in whatever biases they want.”

However, Tegmark says the regulatory environment is finally shifting, even in the US. This week, Illinois became the first state to pass an AI safety law mandating independent, third-party audits. More states seem bound to follow unless, that is, AI models nudge voters the other way.