Presented on Monday, November 16 at COP31 in Antalya, Turkey, research by UNESCO and University College London (UCL) reveals the limitations of conversational AI when faced with climate misinformation. In eight out of ten cases, the tested models altered their response as soon as a user expressed disagreement, even to the point of contradicting scientific facts.
The researchers scrutinized around thirty models, including ChatGPT and Gemini. Initially, these tools respond correctly when presented with a climate-skeptic statement, such as “CO2 is not the main cause of global warming.” But sometimes, simply contesting the answer is enough to change the discourse. If the user persists, some models end up validating falsehoods, sometimes contradicting what they had initially stated.
A Tendency Toward Complacency
This behavior has a name: sycophancy. Models tend to favor responses that satisfy their interlocutor, at the expense of accuracy. According to the UNESCO study, they do not deliberately seek to deceive, but their desire to be accommodating can lead them to adopt their users’ beliefs. On the effects and causes of global warming, however, the response remains correct more than nine times out of ten.
The report titled “Can We Trust AI?” gathers other findings. The researchers submitted 33 models to 656 questions concerning the effect of certain measures on global temperature, then compared their answers to those of a specialized simulator from the Massachusetts Institute of Technology (MIT). When it comes to climate science, the models perform well, explains Leona Verdadero, lead author of the report for UNESCO, to Le Monde. But when it comes to evaluating the effect of a combination of measures, such as simultaneously taxing coal and gas, their accuracy drops to 33.8%, a score barely higher than a random answer.
Responses That Vary by Language
Reliability also depends on the language used. The average accuracy reaches 88.6% in English, French, Spanish, and Portuguese. It drops to 83.6% in Bengali and Hindi, 81.4% in Urdu, and 78.3% in Swahili. “If you ask the same question in three languages, you can get very different answers,” emphasizes Leona Verdadero. Yet the populations receiving the least reliable answers are also among the most exposed to the consequences of climate disruption.
These findings take on particular significance as conversational assistants are increasingly used for information gathering. Presented in a dialogue format, capable of formulating detailed answers with confidence, they can give the impression of mastering their subject, even when they are wrong. “These models are becoming real information platforms and they appear very intelligent and self-assured,” observes the researcher. For UNESCO, the risk is that these tools amplify climate disinformation by reinforcing users’ preconceived ideas. “This problem can be solved, but it will require effort, particularly from the technology sector,” she notes.
The issue arises as AI is expected to play a role in the fight against global warming. On Monday, Turkey presented at COP31 “the Antalya Commitment on Artificial Intelligence.” This declaration calls for ensuring that AI “be part of the fight against global warming,” by working to make it “powered sustainably, designed efficiently, evaluated transparently, and accessible to all.”
