Are LLMs Re-Coding You?

AI chatbots use language constructed of tokens, trained on billions of parameters. But the mathematical formulas behind them also carry specific worldviews and values. Are machines reprogramming us? An interview with Roberto Simanowski, cultural scientist and media philosopher, author of Language Machines: A Philosophy of Artificial Intelligence (Sprachmaschinen: Eine Philosophie der künstlichen Intelligenz).

By Artur Olesch, 12-minute read

A handful of leading AI chatbots answer questions and give advice to millions, even billions, of people across different cultures and value systems. What’s wrong with that?

When a handful of companies determine the “worldview” of a technology that answers questions on moral and political controversies for a global audience, they expose themselves to the charge of “Western ethnocentrism” and “missionary moralism” – that is, if they favor the value system of the West and ignore the perspectives of other states and cultures.

Representatives of the “Global South” rightly see this as a continuation of epistemic neocolonialism through technology and protest in papers such as the AI Decolonial Manyfesto.

But the problem runs deeper. It leads back to the heritage of the Enlightenment and modernity’s self-understanding, i.e., to the question of whether there are universal values at all and only one concept of progress applicable to all cultures.

Are there some universal values that chatbots should represent, and who gets to decide which ones are right? Or is the whole idea of a one-size-fits-all chatbot simply wrong?

Insofar as there is no single value system that is universally valid, there can also be no chatbot that represents such a value system. But it is not enough either that different chatbots represent different value systems, for in that case everyone could and probably would still have their own chatbot with their own value system.

What matters is that one is exposed to different value systems, as has hitherto been the case when one reads books from different cultures. For this diversity to be mirrored in a chatbot, the chatbot would have to offer, after answering a question, to answer that same question from the perspective(s) of another religion or philosophy.

AI companies use very different methods to shape their chatbots’ moral character – Democratic Public Input, the Helpful—Honest—Harmless triad, Anthropic’s Constitutional AI, OpenAI’s Model Spec. If even the companies building these systems can’t agree on an approach, will we ever reach consensus on how AI should behave?

These are, to be sure, different methods companies use to shape the moral character of their chatbots, but they do not necessarily rest on different content. It is worth noting that the companies themselves partly reflect on the fact that the value system with which they equip their chatbot is not universally valid.

However, this does not change the fact that this chatbot is then nonetheless distributed globally with precisely this value system.

So should local, highly personalized AI chatbots that understand users’ cultures and values replace the all-knowing chatbots we have today?

With this, we have arrived at the opposite pole of what concerned us before. The locally calibrated or even personalized language machine no longer exposes us to a foreign value system, to be sure, but it may possibly hold us captive in our own ideological bubble. That is not a bit better. Better would be if we continued to be exposed to different perspectives and value systems, as was formerly the case when we read different books on a topic or were otherwise exposed to different views.

European politicians often say we need AI that represents “European values.” What are these European values, exactly?

It’s not obvious what “European values” actually refers to. What is clear is that European politicians want AI independent of the United States. But “European values” is itself an unsettled category. On issues like same-sex marriage and LGBTQ rights, Germany and Hungary stand worlds apart. Before Europe can build an AI that represents its values, Europeans first have to agree on what those values are.

You argue that chatbots talk us into opinions we don’t actually hold. But that’s always been true of opinion leaders, books, media, TV. What’s different about a machine that can analyze and synthesize millions of points of view and present the one that is an outcome of mathematical calculation?

The problem is that the language machine is precisely not an ordinary communication partner, even if it may sometimes appear to be one. And the problem does not lie in the fact that it is an artificial communication partner. The problem lies in the fact that it is not a communication partner at all, but a communication machine.

When I speak with a language model, I do not speak with a human being or their avatar about God and the world. I speak, through the language model, with a vast number of people whose views on God and the world the language model has absorbed and passes on to me in a certain mixture. The language model becomes the mouthpiece for all those people whose utterances it has absorbed into its training data, not for each one in turn, however, but for all of them simultaneously.

This is what makes communication with the language machine so peculiar: it places itself between me – between us – and all those people it represents; it decides how those people will speak to me, to us. Only through this does the question arise: which value system should the language machine be equipped with? Communication partners can of course represent their respective value systems, to which I then have to relate wherever I encounter them: in a conversation on the street, on a social network, in a public lecture, in a book.

Now, however, the language model administers all these potential communication partners and brings them into contact with me, with us, only through its own “voice” and according to its own standards.

A chatbot may be able to change people’s minds more effectively than a human can. Vaccination decisions are one example. Machines bring the right arguments in a surprisingly persuasive way. When hard scientific facts are involved, should a chatbot act as a moral machine, or stay strictly objective?

In September 2024, an article in Science attracted a lot of attention: in an experiment, an AI turned out to be more successful than a human at talking conspiracy theorists out of their beliefs, and this precisely as an AI. The fact that it proved more successful than a human conversation partner was explained by its being a better listener and better informed.

These are arguments one unfortunately has to factor in nowadays, since these basic rules of communicative decency can hardly be taken for granted anymore.

Still, the question remains whether it was really the AI’s more patient listening that made the difference. Was it not rather the fact that the AI, not being a human, has no political agenda to pursue but is committed solely to the truth?

I consider it very likely that it was precisely the AI’s quality as a soulless interlocutor that lent it additional authority in this moment. In this sense, the AI could also provide information and advice on medical questions, based on scientific findings that it, after all, takes in much faster and more comprehensively than any human ever could.

We often discuss AI ethics through examples like a self-driving car choosing to drive into a child and a group of elderly people, or a triage system deciding between prioritizing hospital admission for an 80-year-old vs. a young mother. But if humans don’t have a settled ethical answer to these dilemmas, why should we expect a machine to have one?

The problem is that we have to make such a decision the moment we hand it over to the algorithms, or rather, to artificial intelligence. Up to now, in the event of an accident, the driver reacts spontaneously, which cannot really be described as a rational or even “cold-blooded” decision.

A self-driving car, however, operates with if-then rules that it either learns through observation or that must be programmed into it. Here, then, the triage decision must be made in advance. This is the ethical challenge that the new technology poses to society. Put differently: technological progress forces society to settle ethical questions for which it previously had no answers – or which it deliberately preferred not to answer.

Can we ever find consensus on such hard ethical issues? And if not, what’s wrong with letting an algorithm decide?

There is a chapter in my book called “Mathematical Ethics” that pursues precisely this question. Since AI can analyze existing data better than a human and simulate future scenarios on that basis, it is only natural to let it decide what is best for all of us. Then it would be the AI that optimizes the social system of human beings – just as, in the context of cell therapy, it will soon optimize the human immune system through a tailor-made protein design.

One need only redirect the available computing power for data processing accordingly: from the molecular secrets of life to the social ones. This next step is one that some moral philosophers have already taken. Interestingly, it is sometimes precisely the global challenges facing humanity that serve as the argument for handing over power to the AI’s central regime, on the grounds that only AI is truly equal to this task.

Humboldt argued that language shapes thought. But every person already has a set of values and beliefs before they ever talk to a chatbot. Can language alone really change them?

Of course, we all already have certain values and beliefs, but to the extent that these can be influenced through communication, and as our communication increasingly takes place with chatbots, these can also alter our perspectives.

This holds especially when we always communicate with the world through the same chatbot with the same value system. The repetition of a particular thought is the precondition for our self-referential cognitive system to open itself to it.

New studies suggest around 50% of people treat chatbots as friends – the percentage varies by culture – and some even say they miss them. What does that tell us about society today, about us?

Given the “loneliness epidemic” that has been spreading for some time now, not only in the United States, it hardly comes as a surprise that AI has moved from assistant to companion.

Another keyword for understanding this phenomenon is the economy of attention as a consequence of the flood of information on the internet: AI companions are so sought-after not only because they never contradict, but also because they will talk to one at all; and do so with a patience that, in an age of scarce attention and increasing loss of concentration, can hardly be expected of human beings anymore.

A third keyword is snowflake: in 2016, the word of the year for a generation that came of age in the 2010s and was considered less resilient and more easily offended than earlier generations. The psychologist Sherry Turkle writes in her 2024 essay “Who Do We Become When We Talk to Machines?”: “The desire to sidestep vulnerability links the field of artificial intimacy to studies that suggest a near epidemic of emotional fragility. […] The assumption of fragility, the call for ‘safe spaces’ in school and the workplace, and the idea that coursework should include trigger warnings when presenting challenging material – find an odd echo in the comments of chatbot users on Reddit, such as, ‘My Replika won’t hurt me the way people will,’ or ‘I’m safe with my Replika in a way that I’m not with real people.'”

Trust between doctor and patient usually grows out of vulnerability and human contact. Can a machine really build that kind of trust, even if it’s only a simulation of trust?

I do not believe that the machine can replace a psychologist or a therapist. It cannot even replace a friend, because it does not really care about the other person and, unlike a therapist, does not have the other’s interests in view but rather the AI company’s self-interest: to keep the user in conversation for as long as possible.

Setting aside this question of trust, it should nevertheless be noted that one also hears that AI reads a human being – his choice of words and his nonverbal signs of communication – better than another human can, and in a certain respect is thus perhaps even the better therapist. Quite apart from the fact that the chatbot is always available whenever the person needs advice.

Does it actually matter whether we get advice from chatbots or humans, whether political decisions are made by humans or AI, or whether these interview questions were written by AI or by me?

It really does not matter whether these interview questions were written by the AI or by you, as long as I can make something of these questions, and as long as you in turn can then make something of my answers.

And it is the same with the chatbot’s advice: as long as it helps us – that is, meets our expectations – it does not matter that it does not come from a human being. Functionality takes the place of authenticity here. Political decisions, too, are rational decisions and can be made by the AI just as well as, if not even better than, by a politician.

All of these are text types that are content-oriented, not sender-oriented. It is different with opinion pieces, love letters, or letters of congratulation, which express a personal stance on the part of the writer. In these texts, the human being must be present, for only they can truly feel the feelings such texts intend to produce in the reader. An AI knows feelings only from its training data and, when it expresses them, does not know what it is talking about.

In your view, what is the single most dangerous feature of chatbots today?

That we forget they are chatbots.

Thank you for your time!

---