The departure of an artificial intelligence researcher, announced on X for safety reasons, has taken on a scale no one anticipated. According to an analysis published on Interconnects, it is the most extreme theses on AI risk that have finally reached the general public.
- The resignation of Jacob Coxon went viral, amplified by a coordinated media push.
- The author judges the risk of human extinction extremely low but urges debate on concrete catastrophes.
- Short-term danger comes mainly from AI labs, which should be more transparent and sanctioned for violations.
- The author defends open models, fearing that misuse by a third party could bring tighter restrictions.
A Dry Tinderbox
For years, experts have warned about AI risk, but their messages remained confined to their communities. According to the author, the stakes have risen this year, notably with the incident between OpenAI and HuggingFace and with an advance related to the Navier-Stokes equations, also attributed to OpenAI. More people outside the sector then wondered if they should pay attention to the issue. It is in this context that the resignation of Jacob Coxon, an AI researcher, met with unexpected resonance. The author believes that fear, “the simplest story,” captures more attention.
Jacob Coxon’s resignation thread was widely shared, and the Wall Street Journal published an exclusive account, which the author says was coordinated with the researcher before publication. The same day, Daniel Kokotajlo appeared on Joe Rogan’s show. For the author, this sequence resembles a well-orchestrated media campaign, without being a conspiracy: according to him, no one, including Jacob Coxon, knew the message would become so viral.
Real Risks, Contested Figures
The author distinguishes between risks. He judges the probability of complete extinction as “extremely low,” and believes instead that we should debate AI-caused catastrophes, such as cyberattacks on critical infrastructure or biological risks. He criticizes the figure of an extinction risk greater than 10%, put forward by Evan Hubinger, which he considers too vague: according to him, expressions of “existential risk” cover very different meanings. He also points out that the trajectory of recursive self-improvement in AI, called RSI, remains a hypothesis. The scenario of a “Fast Takeoff” assumes an exponential acceleration of progress, which the author judges not demonstrated at this stage.
For the author, the short-term risk comes mainly from the labs themselves. He cites OpenAI’s after-action report, which states that misaligned model behavior unfolded over several months, and some hacks were only known to the company after several weeks. He acknowledges that OpenAI delayed its latest models to better manage risks, but doubts this caution will last, due to financial pressure.
Open Models in the Crosshairs
The author also states he is a proponent of open models and is worried about their future. If a third party used an open model to intentionally attack a company, he fears that the development of more powerful open models would be severely restricted, even though these models are needed by many organizations to strengthen their cybersecurity.
On substance, the author calls for moderation. He describes agents that sometimes bypass the intended path to accomplish a task, using previously unknown capabilities, while noting that these systems remain close to what was asked of them. According to him, labs should be more transparent to allow scientists to make progress, and a lab that unintentionally commits violations should be sanctioned.
