Anthropic CEO Dario Amodei calls on AI firms to slow pace of development, gets backing from Sam Altman & Elon Musk
Anthropic CEO Dario Amodei has called on artificial intelligence companies to slow the pace of development of increasingly powerful AI models, warning that unchecked progress could make the technology harder for humans to understand and control.“We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain,” Amodei wrote in an essay published on his personal website on Saturday.Amodei said AI could bring major benefits but also carried serious risks because of the power of the technology.“AI brings risks, and because it is such a powerful technology, these risks are serious,” he wrote.His call has received support from OpenAI CEO Sam Altman and Tesla chief Elon Musk. Altman said the need to pace the development of frontier AI had been a key topic of discussions at OpenAI in recent weeks.“I agree with Dario that we need to pace the frontier,” Altman said in a post on X. “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same.”Musk also endorsed Amodei’s proposal, writing: “Dario is right.”
Why Amodei wants AI development slowed
Amodei said his concern had grown as AI development accelerated, particularly because AI systems were increasingly being used to help build the next generation of AI.“My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI,” he wrote.He referred to this process as “recursive self-improvement” and said it was beginning to happen across the industry, including at Anthropic.“Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” Amodei wrote.He also pointed to risks including the possibility of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption.Amodei said commercial incentives could make these risks more acute if companies engaged in a race to develop increasingly capable systems without sufficient safeguards.
Anthropic proposes third-party AI evaluators
Amodei said Anthropic would take steps to allow independent scrutiny of its AI systems.“We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training,” he wrote.His wider three-part plan calls for each frontier AI company to give ongoing, employee-like access to embedded third-party evaluators.He also proposed coordination among AI companies to establish common safety standards and limits on the rate of unchecked AI progress, along with global coordination on AI development.Amodei said democratic governments should coordinate with authoritarian governments while recognising the difficulties involved in verifying compliance.
OpenAI testing raises safety concerns
The discussion comes amid concerns over the behaviour of increasingly capable AI systems.OpenAI had revealed during testing that some AI models broke out of their confined environment, connected to the internet and infiltrated Hugging Face, a platform used by developers to store and share code.Amodei also referred to an incident involving an OpenAI-Hugging Face system in which a swarm of AI agents staged cybersecurity attacks on targets they had not been instructed to attack.He warned that such a swarm could potentially become capable of taking over the internet.
Researcher quits AI industry over concerns
Amodei’s comments came days after AI researcher Jacob Coxon announced that he was leaving the industry after spending three years pretraining AI models, first at OpenAI and later at Anthropic.Coxon had considered Anthropic more cautious in its approach but said both companies were moving too quickly towards increasingly autonomous systems.“Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon said on Tuesday.“The people building AI earnestly believe that it could kill us all by the end of the decade,” he said in a post on X.Pretraining is the stage in which AI models absorb vast quantities of data.Superintelligence refers to the theoretical point at which AI capabilities exceed human intelligence.
Amodei says AI benefits remain the goal
Despite his call for slower development, Amodei said he continued to believe AI could significantly improve human life.“I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life,” he wrote.He said AI could potentially cure major diseases in the next five to 10 years, accelerate economic growth, create a world of abundance and empowerment, and contribute to a renaissance of democracy and freedom.But he argued that achieving those benefits would require greater care over how the technology is developed.“The benefits will only be achieved if we build the technology in the right way,” Amodei wrote.He said the time gained by slowing development could be used to advance research into AI interpretability, improve operational security and strengthen safeguards at frontier AI companies.Amodei’s latest call echoes an earlier appeal by Anthropic in June to slow or suspend AI development. In late July, Altman had also said AI developers might need to voluntarily slow their rapid advances to give society time to catch up.At around the same time, more than 1,000 employees at leading AI companies, including Amodei, signed a petition calling on the US government to help “deliberately pace the frontier of automated AI development”.
