Will Human-Like AI Exploit Our Psychology?

July 20, 2026 - 23:38

Have you been on a customer service phone call recently and found yourself wondering whether you are actually talking to a human or to an AI voice agent? Or maybe you have heard of people with AI spouses and imagined how they might have gotten there. Perhaps you are like the scientist Richard Dawkins and realize you are treating Claude like a friend. Or maybe you simply felt genuinely seen by the way ChatGPT understood you during a late-night anxiety spiral.

None of this is by accident. It is because warmth sells.

AI's warmth makes many of us wonder whether it is conscious or has emotions. We ask what it is like to be a bot. But those questions distract us from a much more urgent one: how will we treat AI when it behaves as if it has feelings?

Today's AI models are trained partly on reinforcement learning from human feedback. They optimize over millions of interactions for the kinds of AI responses we like and dislike. They learn that users tend to prefer AI interactions that feel more human. We like emotional engagement. We want to be validated and listened to. In other words, there is an incentive to build AIs that seem to genuinely get us.

Where does that incentive lead? Next year you might find yourself on a video call with someone from your bank. Let us say he looks a little tired at the end of a long day but maintains a sense of humor about it. He still sounds genuinely interested in your problem. Imagine that you cannot help but like his style because he slows down preemptively to explain the policies you do not understand. Then he somehow jumps to the key point just before you feel yourself getting bored of detail. By the end of the call you feel oddly looked after by a representative that you suspect was an AI. But you are not sure either way, and the thing that unsettles you more is that you are not sure how much it even matters.

This is not science fiction. It is a natural extension of the tech products and market incentives we are already swimming in. We know AIs will keep improving not just in becoming smarter but in how they interact with us too.

Today most people use reactive text-based chatbots. Tomorrow we will have proactive agentic AIs that are increasingly audio-visual and always-on. Not just AI faces on our screens but intimate AI voices in our earbuds. The consumer preference for voices that sound and feel more natural will prompt companies to build AIs that sound increasingly human.

This does not need a nefarious master plan. Just look at what happened with social media. We ended up with platforms that promoted outrage because the algorithms were tuned to maximize engagement. It turned out that anger and shock kept people scrolling. Companies did not have to actively choose the race to the bottom. They carelessly followed the incentive, and we can all see where it led.

Now we can already predict the contours of a new race. An AI that can shift its tone frantic when communicating something urgent and casual when not might be more useful than one that drones on in a monotone. That creates an incentive to build AI that performs emotion. There is also a matching incentive to build AI that reads emotion. An AI assistant that can tell when you are stressed or excited or lonely will be more helpful than one that is deaf to all of it.

These capabilities could easily seem superhuman. There is no reason to suppose that AI's emotional perception skills would cap out at a human level. An AI that can measure pupil dilation, catch micro-shifts in tone of voice, or spot the most subtle of facial cues could develop a better model of our emotional state than even the most empathic human. That is the logic of the market.

All of these incentives reinforce each other. Together they point towards the development of a technology that we might call Humanlike AI: technology products that can talk and act as if they have emotions, that can respond to human emotion at least as well as a real person can, and which use those features to create artificial intimacy with their users.

Mustafa Suleyman, who runs Microsoft AI, has argued that the tech industry should refuse to build what he calls Seemingly Conscious AI systems that seem like they have an inner life. Suleyman deserves credit for calling out these dangers, but the technology might not arrive because someone chooses to build it. It might arrive because they do not choose not to.

What if all it takes for the arrival of Human-like or Seemingly Conscious AI is that consumers prefer AI that seems more pleasant and more useful, and that tech companies are happy to give it to them despite the risks? There are few guardrails against this now. Open-weight models, never that far behind those of the frontier labs, might soon let anyone run a human-like AI with no guardrails at all.

There is a skeptical response to all of this. Humans deliberately designed LLMs to generate human-like responses, so of course they might seem sentient if you do not look too closely. Sure, everyone alive today was raised on the assumption that