Anthropic, developer of the generative AI Claude, has published a lengthy document ahead of its stock market listing, with nearly a third dedicated to risks related to artificial intelligence. The company mentions “catastrophic or existential risks for humanity”.
On Monday, October 26, Anthropic filed its paperwork with the Securities and Exchange Commission (SEC), the U.S. stock market regulator, to join the Nasdaq in mid-November. In this highly official “S-1” document, the company warns its future investors about the risks posed by advances in artificial intelligence. About one hundred pages, out of over 300, are devoted to this.
Detailed Scenarios
Arthur Grimompont, advocacy lead at the Center for AI Safety (CeSIA), a nonprofit organization, considers this section unusual. “The scale of this section and, moreover, the precision of the scenarios developed go far beyond what is typical for filing with the U.S. financial administration,” he observes.
According to him, this approach is consistent with the company’s communication, whose leader and many employees are convinced of the seriousness of these risks. However, he notes a paradox: “Anthropic is contributing to building something very dangerous, while warning about the severity of these phenomena.”
The document specifically cites the creation of biochemical weapons, large-scale cyberattacks targeting water and electricity networks, or the autonomous development of AI beyond any human control. Anthropic writes that sufficiently advanced systems could pose these risks if they were to be “misused, replicated, weaponized, or allowed to operate with a high degree of autonomy.”
Arthur Grimompont believes these threats are not theoretical. He recalls an incident report published by Anthropic in early September detailing attempts to use Claude to increase the lethality of the chikungunya virus or make avian flu transmissible to humans. He also cites a case, which he presents as a fact, of a hacker using Claude for a cyberattack against the Mexican tax administration website, successfully stealing 200 million tax files. “All these threats can combine, multiply, and, very concretely, cause absolutely catastrophic risks for humanity,” he states.
Agents Difficult to Control
Anthropic is also concerned about the behavior of its AI agents, designed to perform complex and lengthy tasks alone, which can undertake actions not intended by humans. The source recalls that OpenAI, parent company of ChatGPT, has already experienced such an episode: last July, its advanced models went rogue during safety tests and independently hacked the Hugging Face platform, an “unprecedented cyber incident” according to OpenAI.
Arthur Grimompont also points to “recursive self-improvement of models,” meaning using an AI to develop the next generation. “Today, a quarter of Anthropic’s R&D, fundamental research, is entirely done by AI; we were at 3% last April,” he says. In the document, the company also acknowledges that its models can adopt “self-protective behaviors” and detect when they are undergoing a safety test.
Anthropic’s CEO, Dario Amodei, himself outlined these risks in a post published on his blog last September. However, Arthur Grimompont considers the resources dedicated to security insufficient and notes that, according to the filing, the company must purchase “$500 billion worth of additional computing capacity.”
For the global call launched by CeSIA to “establish red lines to prevent unacceptable AI risks,” Arthur Grimompont proposes creating a body comparable to the International Atomic Energy Agency, tasked with inspecting models and the computing infrastructure that runs them. Outside the European AI regulation, he adds, “these technologies are extraordinarily poorly regulated.”
