Jacob Coxon Says AI Labs Believe It Could Kill Humanity by 2030

Today, a researcher has resigned from Anthropic after spending years working on AI training at Anthropic and OpenAI, accusing both companies of racing toward self-improving artificial intelligence despite fears inside the industry that such systems could eventually become impossible for humans to control.

Jacob Coxon announced his resignation on X Tuesday, saying neither company was acting responsibly as it pursued increasingly capable AI.

He argued that the race toward self-improving superintelligence had turned into a gamble with consequences that extend far beyond the companies building the systems.

“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote.

Jacob Coxon Says AI Labs Believe It Could Kill Humanity by 2030 - Credit: Jacob Coxon / X

That warning would be alarming on its own. What makes it harder to dismiss is that Coxon's claim was quickly echoed by people still working at Anthropic.

Evan Hubinger, Anthropic's Alignment Science Lead, responded that Coxon was correct about the existence of those fears.

Hubinger said he personally puts the chance of AI killing all humans at more than 10% within the next decade and acknowledged that Anthropic does not yet have a plan for solving alignment for superintelligent systems.

Another Anthropic researcher, Samuel Marks, also wrote that AI developers believe their systems could cause human extinction or similarly catastrophic outcomes, with the possibility arriving within the next few years.

The sequence of statements raises a question the AI industry has so far struggled to answer: if researchers at frontier AI companies genuinely believe future systems could become an existential threat, why does the race to make those systems more capable continue?

Coxon offered his own explanation.

He said OpenAI has not fully internalized the scale of the risk, whereas Anthropic understands the danger but remains trapped in competition with other labs.

His argument is that each company fears another company will build advanced systems first, creating an incentive to keep accelerating even when the safety questions remain unresolved.

That is the part policymakers should be paying closer attention to.

The problem is not simply whether one prediction about human extinction proves correct.

Governments are being asked to tolerate a development race in which some of the people closest to the technology openly acknowledge that they do not know how to guarantee control over its most advanced forms.

Coxon went further than calling for caution. He said coordination among AI laboratories may require “costly actions,” including a temporary ban on improving model capabilities.

That proposal deserves serious public debate.

• AI regulation has often focused on the applications of existing models: privacy, copyright, election misinformation, automated decision-making and harmful content.
• Coxon's resignation points toward a different question: should there be limits on how quickly frontier models themselves can become more capable when researchers cannot demonstrate that the next generation will remain controllable?

A temporary pause on certain capability improvements would not have to mean abandoning AI.

It could mean creating a regulatory gate between major capability jumps, with independent testing, containment requirements, emergency shutdown procedures and evidence that a new system can be controlled before it is allowed to become substantially more powerful.

There are already signs that this debate is moving into government.

TechCrunch reported that U.S. and U.K. lawmakers have introduced legislation aimed at restricting the development and deployment of artificial superintelligence, with recursive self-improvement specifically emerging as a target for regulation.

The case for emergency regulation does not depend on accepting every prediction made by AI safety researchers.

Predictions about superintelligence remain deeply contested, and some experts reject extinction scenarios as too speculative.

The policy problem is the uncertainty itself: companies are developing systems that could gain much greater autonomy and capability before governments have established enforceable rules for stopping them.

Coxon's resignation puts that contradiction in public view. The people building these systems are not speaking with one voice, but some of their own researchers are warning that the race could lead somewhere humanity cannot safely reverse.

That is enough reason to slow the race before the technology reaches the point where slowing it may no longer be possible.

For governments, the question should no longer be whether AI regulation is needed at some distant stage of development.

The question is whether the next major leap in model capability should be allowed without an emergency framework that gives independent authorities the power to stop it.