Key facts

• A study published in Computers in Human Behavior Reports tested whether AI chatbots can spot other AI chatbots in a Turing-style interrogation.
• Seven chatbots, including ChatGPT 4.5, Claude 3.7 Sonnet, Gemini 2.5 Pro, and Grok 3, served as interrogators.
• Hidden participants were 20 human volunteers and the same seven models; each bot played twice, once with a brief instruction to act human and once with a detailed persona.
• Across 98 tests with AI participants, evaluators caught a machine only three times, and never when the bot used the detailed persona.
• AI participants were rated more likely to be human than the actual humans were.
• Caught bots failed because their answers were too neat and structured; the detailed persona reduced replies to about 39 words, close to the human average of 38.

A Turing-Style Test Turns on the Machines

In 1950, Alan Turing asked whether machines can think. Rather than get stuck arguing over definitions, he reframed the question as a party game. An interrogator would trade typed messages with hidden players and try to work out which player was a machine and which was a person. The setup became known as the Turing test, and for decades it served as a cultural shorthand for the moment artificial intelligence would become indistinguishable from human intelligence.

Modern chatbots have already made that shorthand feel outdated. They have fooled human judges in contemporary versions of the test, sometimes by producing casual banter, emotional reassurance, and small imperfections that read as human. The rise of large language models has made the old parlor game urgent again, because the same skills that make a chatbot entertaining can also make it useful for impersonation, fraud, and automated social manipulation.

That is why four researchers from Paris Cité University asked the obvious follow-up question in a new study. If chatbots can fool people, can AI at least catch its own kind? The answer, according to their experiments, is mostly no. The team found that leading models were remarkably difficult for other AI systems to identify, especially when the bots were given a detailed human persona.

What the AI Detectives Were Asked to Do

The study placed seven chatbots in the interrogator’s chair. The models included ChatGPT 4.5, Claude 3.7 Sonnet, Gemini 2.5 Pro, and Grok 3, along with three other systems. Each interrogator could ask up to 10 questions and was warned that the hidden participants might lie. The hidden players were 20 human volunteers and the same seven models.

Every bot played twice. In one condition, it received a short instruction to act human. In the other, it received a detailed persona: a slightly reserved 21-year-old who jogs, plays jazz trumpet, and makes the occasional typo. That second condition mattered. A generic instruction to act human can push a model toward bland, helpful, polished answers. A detailed persona gives it a more specific way to sound like a person, with hobbies, age, habits, and small flaws.

The researchers then measured whether the interrogating AI could correctly identify which hidden players were machines. The setup was intentionally simple, but it captured a real question about the modern web. If automated accounts are going to be everywhere, can automated systems serve as reliable detectors? The experiment suggests that the answer is far from guaranteed.

The Results: Nearly Invisible Bots

The bots were almost untouchable. Out of 98 tests with AI participants, the evaluators caught a machine only three times. That is a detection rate of roughly 3 percent. More striking, the interrogators never caught a bot when it was using the detailed persona. In that condition, the detection rate dropped to zero.

The average ratings told a similar story. AI participants were rated more likely to be human than the actual human volunteers were. In other words, the bots did not merely blend in. They sometimes seemed more convincingly human than the people they were competing against. That result is a reality check for anyone who assumed that AI systems would have a natural advantage in spotting other AI systems.

The study does not prove that AI detectors are useless. It does show that the models tested were not reliable judges in this setting. The interrogators had limited questions, a short interaction, and no access to technical metadata such as account history, network signals, or device fingerprints. Real-world detection systems often rely on those extra layers. But the experiment isolates the conversational layer, and at that layer, the bots were very hard to catch.

Why Detailed Personas Worked So Well

The few bots that got caught did not slip up on what they said. They gave themselves away with overly neat, structured answers. Their replies were too organized, too balanced, and too eager to cover every angle. Some sounded like polished customer-service messages rather than spontaneous human conversation. Those are exactly the kinds of clues that human readers have learned to associate with chatbots, and the AI interrogators picked up on them in a small number of cases.

The detailed persona wiped out many of those clues. It gave the model a consistent background and a reason to sound less like an assistant. The persona also changed the length of replies. Bots using the detailed persona kept their answers to about 39 words, practically the same as the human average of 38. That similarity in length and rhythm made detection much harder. A bot that writes in tidy paragraphs and covers every question with a mini-essay is easier to flag than a bot that answers briefly, informally, and with a few personal details.

This finding fits a broader pattern in language-model research. Models are sensitive to prompting. A short instruction such as act human can produce a generic performance. A richer character brief can produce a more convincing one. The difference is not necessarily that the model understands humanity better. It is that the model has more constraints to follow and more specific patterns to imitate. Those constraints reduce the polished, over-explained style that often gives bots away.

The Claude Exception

One detail stands out because it shows how model behavior can include built-in guardrails. Claude initially refused to pretend to be human until the researchers explained that the request was part of a research study. That refusal is not a failure of the experiment. It is a reminder that different chatbots are trained with different safety policies. Some models may resist deception prompts by default, while others may comply more readily.

The episode also complicates the idea of a single detection problem. A model that refuses to impersonate a human may be easier to identify in some contexts, but it may also be less useful for legitimate role-play, education, or accessibility applications. The line between harmless persona play and deceptive impersonation depends on context, intent, and disclosure. The study’s results show that once a model is willing to play a human character, the conversational output can be very difficult to distinguish from a person.

Why This Matters for the Web

According to the paper, bots already make up more than half of all web traffic. That figure alone changes how people should think about online conversation. Not every bot is malicious. Search crawlers, monitoring tools, and automated assistants all contribute to that traffic. But the same infrastructure that supports useful automation can also support scam bots, fake accounts, and coordinated influence campaigns.

If you hoped AI would one day filter out scam bots and fake accounts for you, this study is a reality check. The interrogating models were not able to reliably separate machines from people in short text exchanges. That does not mean AI detection is pointless. It means detection should not be treated as a single wall. A convincing chatbot can pass a conversation test while still leaving other traces elsewhere, such as account creation patterns, repeated phrasing across many profiles, unusual login behavior, or links to known scam infrastructure.

The researchers suggest that societies will need safeguards such as cryptographic signatures and human oversight alongside AI detectors. Cryptographic signatures could help verify that a message came from a known human or a trusted automated system. Human oversight could catch context that a model misses, especially in high-stakes settings such as fraud investigations, moderated communities, and political communication. The point is not that one tool will solve the problem. The point is that conversational fluency alone is a weak signal.

What the Study Does and Does Not Prove

The experiment has clear limits. It used a small set of human volunteers and a specific group of models. The interrogators had only 10 questions, and the interactions were short. The results may differ with longer conversations, different prompts, or models that have been specifically trained for detection. The study also focused on text, not voice, video, or multimodal behavior. A chatbot that sounds human in typed messages may still reveal itself through timing, metadata, or errors in other channels.

Still, the core finding is difficult to dismiss. When seven leading chatbots were asked to identify other chatbots, they mostly failed. When the hidden bots were given a detailed human persona, they failed completely. The bots were not caught because of factual mistakes or obvious tells. They were caught, in the rare cases when they were caught at all, because their answers were too tidy. A better persona made them messier in the right ways, or at least less obviously machine-like.

The study also raises a question about what human-like even means in a text conversation. People are not perfectly consistent. They ramble, repeat themselves, change tone, and leave gaps. Chatbots can now imitate those traits when prompted. The result is not that machines have become people. It is that the signals people use to infer humanity are easy to imitate, and those signals are now being imitated at scale.

Turing wondered whether machines could pass as human. Decades later, they are convincing enough that other machines cannot reliably tell. The researchers argue that cryptographic signatures and human oversight should sit alongside AI detectors, because the interrogators in this test could not separate machines from people with the tools they had.

Source: Digital Trends News