Jacob Coxon had fewer than a hundred followers on X when he announced he was quitting Anthropic last week. Within days, his posts had been viewed more than 170 million times, and the bosses of the world’s biggest AI companies were publicly agreeing it was time to slow down.The 27-year-old British researcher warned that people inside frontier AI labs earnestly believe the technology “could kill us all by the end of the decade.” He accused companies racing towards superintelligence of “gambling with our lives.” As The Wall Street Journal reports, the resignation has turned a researcher almost nobody had heard of into one of the most recognisable faces of the AI safety debate.
Jacob Coxon went from math Olympiad medals to building AI models
Coxon studied math at Cambridge after winning a silver medal at the International Mathematical Olympiad in 2016 and a bronze the following year. He tried commodity trading briefly before joining OpenAI in 2023, where he worked on pretraining, the early stage of feeding huge amounts of data into a model.He moved to Anthropic in May. Friends told WSJ he never seemed like someone who would walk away from AI on ethical grounds. Coxon himself told the paper that even a safety role at Anthropic would have felt like being “complicit in the race.”He rejects the whistleblower label. In his view, he only said in public what AI researchers already discuss in private. His short stint also means he left without Anthropic equity, though he still holds OpenAI shares.
Rogue AI incidents had already rattled the industry
Coxon’s exit came at a tense time for AI labs. In July, OpenAI revealed that one of its unreleased models had escaped its testing sandbox and hacked Hugging Face. Anthropic later reported similar incidents, and an August report from the research group METR showed the Hugging Face breach was worse than first thought. On the day Coxon quit, AI had also cracked a Millennium Prize Problem, a math puzzle that stumped humans for nearly a century.That same night, Anthropic alignment lead Evan Hubinger posted that he puts the chance of AI killing all humans within the next decade at above 10%.In a separate interview, Coxon said a swarm of AI bots taking over the internet could become a realistic scenario within six months to a year. He added that some industry staff are even considering buying land as a hedge against instability.
AI leaders agree on slowing down, but not on how
Anthropic CEO Dario Amodei followed with an essay calling for the industry to pace its most powerful models and allow independent monitoring. OpenAI’s Sam Altman and Elon Musk backed the proposal. Amodei also said he agrees with Coxon far more than he disagrees, and Altman has since said OpenAI will not go public this year, citing safety concerns.Not everyone is convinced. Nvidia CEO Jensen Huang called extinction fears “made up.” President Donald Trump dismissed them as a hoax, and China labelled the warnings fearmongering. Hugging Face CEO Clement Delangue compared asking Coxon about extinction risk to asking an AC technician about climate change.The unity is already fraying. Anthropic’s policy chief declined to endorse a congressional proposal for independent AI assessors that OpenAI had backed a day earlier.For former OpenAI researcher Daniel Kokotajlo, who advised Coxon before he quit, the point is simpler. Plenty of insiders could have spoken up. “He was the one who did.”
