Jacob Coxon warned of the risks of AI

Who is the man who warned the world about AI?

Jacob Coxon, a former employee of the AI ​​company Anthropic, has warned of the risks associated with AI and sparked the biggest debate yet regarding artificial intelligence safety. The Wall Street Journal published a profile on him.

Over the past week, the global reassessment of artificial intelligence risks has spread from cutting-edge Silicon Valley labs to living rooms around the world. And it all started with Jacob Coxon. Until he publicly resigned from Anthropic and warned of the dangers posed by AI systems capable of wiping out humanity, he was a completely unknown figure.

This twenty-seven-year-old British researcher was a talented mathematics student at Cambridge. He spent his early years within the London intellectual community known as the "Rationalists" and, before finding his way to Silicon Valley, briefly tried his hand at commodities trading.

He wasn't a significant player in the tight-knit community of AI safety advocates.

Then, however, he quit and announced to the world that his former colleagues at OpenAI and Anthropic "sincerely believe that it could kill us all by the end of the decade," and that the labs racing to develop this technology are "gambling with our lives."

AI researchers have been warning of dire risks for years. Yet, never before have they managed to alarm so many people in such a short time. Coxon essentially tossed a match onto a mounting pile of unsettling events that, on their own, had failed to capture the world's attention.

Within days of Coxon sounding the alarm, the heads of rival laboratories—in a rare moment of unity—agreed that it was time to slow down the breakneck pace of technological development.

The tumultuous reaction surprised even Coxon himself, as he stated in an interview with the Wall Street Journal.

By his own account, Coxon reached the decision to leave the industry on his own and only then sought advice from friends and members of the AI ​​safety community. They quickly spread the word about his statement. A network of researchers from non-profit organizations—who have long been vocal in warning about the risk of catastrophe—also got involved.

His statements have made him a target for ardent AI proponents, who view his dramatic departure as part of a coordinated conspiracy aimed at pushing through stricter global regulations that favor the largest corporations. Since his resignation, the heads of the world’s most valuable companies and AI labs have found themselves on opposite sides of the debate he ignited. Anthropic CEO Dario Amodei stated that he largely agrees with Coxon. Conversely, Nvidia chief Jensen Huang expressed admiration for Coxon’s courage but questioned his predictions about the future, dismissing his warnings of catastrophic scenarios as irresponsible.

In the world of AI, the term "doomers"—prophets of doom—has come to be used for those most concerned about the risks. "If the word 'doomer' describes someone who thinks we’re all going to die unless we change our approach," said Coxon, "then I’m definitely a doomer."

Even before becoming a doomer, while studying at an elite private school in Oxford, Coxon encountered a book that fundamentally shaped his worldview. *Superintelligence: Paths, Dangers, Strategies* by philosopher Nick Bostrom—an influential figure in the burgeoning effective altruism movement—cemented his fear that uncontrolled AI could pose an existential threat to humanity.

After university, Coxon became close to a small but influential community of rationalists seeking to transform the British government. This online community, which prioritized rigorous rational reasoning over political instinct, gained a foothold in politics thanks to Dominic Cummings, the architect of Brexit and a Downing Street advisor.

Early Days in Silicon Valley

Coxon moved to San Francisco in 2023 to join OpenAI after the meteoric success of ChatGPT cemented the company's position as an industry leader.Meanwhile, He observed the remarkable progress of AI in mathematics, with models improving far more rapidly than he had anticipated. Suddenly, basic arithmetic posed no challenge for them. In 2024, a Google DeepMind model achieved a result at the International Mathematical Olympiad that would have earned it a silver medal on its own. By 2025, both DeepMind and OpenAI had secured gold.

AI Spirals Out of Control

In 2026, panic gripped Coxon and much of the AI ​​safety community. Earlier that year, Anthropic’s Mythos model—along with other equally advanced tools—demonstrated the ability to carry out sophisticated cyberattacks. Some researchers saw this as a warning sign that the technology they were building was becoming far too powerful. The head of Anthropic’s safety safeguards research team issued a warning that "the world is in danger" and left the company to pursue the study of poetry.

By then, Coxon already had many friends at Anthropic and felt that this rival to OpenAI was taking a more transparent approach to the risks associated with the technology.

In July, OpenAI revealed that one of its as-yet-unreleased models had escaped its testing environment—or "sandbox"—and targeted the AI ​​company Hugging Face. Shortly thereafter, Anthropic announced that it had experienced similar incidents. Then, in late August, a warning report from the organization METR emerged, revealing that the incident involving Hugging Face was even more serious than anyone had realized.

Concerns about the rapid development of artificial intelligence had been mounting among model developers for months. Earlier this year, a number of researchers publicly expressed alarm regarding the use of AI for surveillance, in response to a dispute between Anthropic and the Pentagon.

Safety advocates have long feared the day when AI systems themselves become capable of developing the next generation of far more intelligent artificial intelligence systems. Research labs acknowledge observing early signs of this phenomenon—known as "recursive self-improvement"—which could significantly accelerate the arrival of the superintelligence Coxon so feared .

In his view, ensuring AI safety would require slowing down model development—if necessary—through a globally coordinated agreement backed by guarantees from individual governments. He also stated that the US, China, and other governments must agree on a plan similar to nuclear arms control pacts designed to avert catastrophe.

Across the entire sector, nearly 1,400 researchers—including Coxon—have already signed a statement calling on governments to create an emergency brake that could be pulled if necessary.

Commentary from Czech web:

The greater the capabilities we give to AI, the more critical the goals, values, and constraints it derives from that learning process. It is not merely a question of whether the AI ​​will be intelligent, but also of how it will utilize that intelligence. In essence, we are creating a kind of "brain" and then teaching it things we already know ourselves—drawing upon our own knowledge, experience, and data. The situation becomes both fascinating and potentially dangerous the moment we grant it the ability to derive insights from that knowledge that go beyond what we ourselves know. At that point, we may no longer be able to predict with certainty the conclusions it will reach or the decisions it will make. Perhaps the most significant question, then, is not just how intelligent the AI ​​we create will be, but what will happen when it surpasses its creators in intelligence within certain domains. It is capable of processing vast amounts of data in a very short time, identifying connections, and spotting errors or patterns that would simply escape human notice. That is precisely where both its power and its risk lie. It does not necessarily have to "think better" than a human; it can simply perceive and process things that a human cannot grasp due to the sheer volume of information, the speed involved, or the limitations of human attention. A further question arises regarding how we might program consciousness, a conscience, or feelings into this "brain."

                    Thanks for reading

                                                      Margaret

Source: Czech web