Researchers at the Wharton School have given a name to a phenomenon many are observing: “cognitive surrender”. In an experiment, participants followed an AI assistant almost as much when it was wrong as when it was right.
- Participants followed the AI almost as much when it was wrong as when it was right.
- Their trust no longer adjusts to the reliability of the tool.
- Radiologists showed the same automation bias, and expertise does not fully protect against it.
- Time pressure worsens surrender, while financial incentives and immediate feedback help limit it.
The study by Steven Shaw and Gideon Nave is based on three pre-recorded experiments, followed by a synthesis involving 1,372 participants and nearly 9,600 trials. Participants had to answer problems from the cognitive reflection test, which prompt an intuitive but false answer and a reflective, correct one. A ChatGPT assistant was available to them. What they didn’t know was that, depending on the trial, hidden instructions made it give either the correct answer or the intuitive error, accompanied by a short justification.
A trust that no longer discriminates
Participants consulted the assistant a little more than half the time. They followed its answer in 92.7% of cases when it was correct, and in 79.8% of cases when it was wrong. The rate of correct answers rose from 45.8% without AI to 71.0% with a correct AI, then fell to 31.5% with a wrong AI. The confidence of assisted participants was nearly 12 points higher than that of participants working alone, even though half of the answers they received were erroneous.
According to the column’s author, the figure that should raise concern is the gap between 92.7% and 79.8%, which remains small. Following an advisor more reliable than oneself is rational. The problem lies in the adjustment of trust: useful trust increases when the tool is right and decreases when it is wrong.
The authors propose adding a “System 3,” artificial cognition, to the two systems popularized by Kahneman: fast intuition and slow deliberation. They also distinguish cognitive offloading, which involves entrusting a task to a tool while retaining control over reasoning, from cognitive surrender, which involves adopting the system’s answer without examination.
A phenomenon already known in medicine
Medicine has long described this mechanism as automation bias, the tendency to rely on an automated system to the detriment of one’s own judgment. A 2012 systematic review listed aggravating factors: workload, task complexity, time pressure, and trust in the system.
In an experiment published in Radiology, twenty-seven radiologists read mammograms with the help of a purported AI, which sometimes proposed an erroneous classification. When the suggestion was wrong, the proportion of correct readings dropped for all: from nearly 80% to about 20% for the less experienced, and from 82% to 45.5% for the most experienced. Expertise protects, but it is not enough.
According to the author, what changes with large language models is not the mechanism but its power. A conversational assistant delivers a written and justified answer on any topic, and cognitive psychology has long associated ease of processing with a feeling of truth.
What protects, and what is missing
Time significantly worsens surrender. Financial incentives, combined with immediate feedback on errors, increased the proportion of participants who corrected a faulty AI from 20% to 42%, without the majority doing so. A taste for intellectual effort and fluid intelligence protect, while high trust in AI exposes one to risk.
The author cites several design levers: asking for one’s own answer before displaying the system’s, making uncertainty legible, presenting information elements rather than a conclusion, and measuring surrender using cases where the correct answer is known.
The limits of these results are acknowledged. Only one family of problems was tested, under laboratory conditions where AI errors are known. In the workplace, no one necessarily signals that an answer was wrong. As for long-term effects, they remain to be measured: in an observational study, adenoma detection in colonoscopy without AI decreased among endoscopists from 28.4% to 22.4% after the introduction of AI in departments.
