Cognitive scientist Gary Marcus delivered an impromptu intervention during a private meeting preceding an audience with Pope Leo XIV. He declared himself an admirer of the encyclical on artificial intelligence, Magnifica Humanitas, and argued for the withdrawal of AI agents with open access to the Internet.
- Gary Marcus says large language models predict well but reason poorly.
- He calls for withdrawing open-access AI agents from the Internet until they are fixed.
- He compares the situation to the Ford Pinto and supports a product recall.
- He argues that AI systems are not conscious, since their word associations do not refer to the real world.
Introduced as ‘first and foremost’ a cognitive scientist, Gary Marcus was trained at MIT under Steven Pinker. He also founded an AI company, acquired by Uber. He says he agrees with Max Tegmark on several of his recommendations, as well as with Stuart Russell, for whom ‘the current trajectory is untenable.’ He also aligns with Valérie Pisano when she states that choices are often confused with inevitability.
Machines that predict, but reason poorly
According to Gary Marcus, the field’s mistake was believing that mastering a single component of intelligence, prediction, was enough to replicate intelligence as a whole. Large language models (LLMs), systems that calculate the most probable next word, ‘do that very well,’ but reason poorly and follow instructions badly. He cites the case of legal references invented by systems when explicitly asked not to create them.
He praises the work of Yoshua Bengio, who, he says, seeks to design a system ‘safe by design,’ while also believing that both new research approaches and public policies are needed. He also discusses agents, systems capable of autonomously chaining actions. He distinguishes those that perform only a specific task, like GPS, from those that can ‘do pretty much anything on the Internet.’ He notes that AlphaFold has, to his knowledge, caused no harm.
Open agents are, in his view, at the root of current problems. He mentions an incident involving Hugging Face, which he describes as ‘really serious,’ and claims there are about 10,000 incidents of this type. He also cites a cyberattack targeting a refrigeration system in France and attacks in Japan. He notes that OpenAI stated most of these incidents were relatively minor, and says he wonders about those that have not been made public.
A product recall, compared to the Ford Pinto
Gary Marcus believes a product recall is necessary, whether imposed by public authorities or voluntary on the part of companies. He states that agents should be withdrawn from the market until they are fixed. He compares the situation to that of the Ford Pinto, whose fuel tanks could explode, and believes the companies involved ‘knew, or should have known’ the risks. He also mentions, in his opinion, economic considerations, with agents consuming more tokens, which increases provider revenues.
Finally, he addresses the question of system consciousness. He judges that it rests on a confusion between correlation and causation. A system that associates words with each other does not mean these associations refer to the real world. He gives the example of a Facebook bot that claimed to send its children to Stuyvesant, when it has none. He finally cites the encyclical, which states that to be conscious, one must have an experience. According to Gary Marcus, it is a mistake to attribute consciousness to these systems.
