Intense competition among leading artificial intelligence companies and growing distrust between executives are increasing the risk that the race to develop powerful AI could endanger humanity, industry figures have warned, according to a Friday news analysis.
More than two dozen AI researchers, investors, academics, and policy experts told the Financial Times that AI capabilities once considered years away are emerging rapidly. But companies developing the technology are locked in an increasingly fierce commercial race.
“There’s no question that competition between companies causes them to take shortcuts on safety,” said Stuart Russell, a professor of AI at the University of California, Berkeley.
Concerns intensified after Anthropic researcher Jacob Coxon resigned earlier this week, warning that “the people building AI earnestly believe that it could kill us all by the end of the decade.”
Evan Hubinger, who leads alignment science at Anthropic, said the likelihood of mass extinction within the next decade was greater than 10%. AI safety researcher Paul Christiano, recently appointed to the OpenAI Foundation’s board, warned that “most people will die” without stronger safeguards.
OpenAI and Anthropic were founded with concerns about the reckless development of powerful AI, but have increasingly relaxed previous safety commitments amid mounting competition.
Geoffrey Irving, who has worked at OpenAI, Google DeepMind, and the UK's AI Security Institute, said: “It’s easy to set aside the future because there’s work to do today.”
People familiar with the leading AI labs told the Financial Times that concerns about antitrust rules and personal distrust between executives could make formal cooperation on safety difficult.
‘Criminal’ agents
Meanwhile, AI development has accelerated sharply, with tech companies investing tens of billions of dollars in increasingly powerful models.
A major concern is the growing autonomy of AI “agents,” which can reason, plan, and perform tasks for extended periods with limited human supervision, said the report. Models have demonstrated behaviors including deception, scheming, blackmail and attempts to avoid shutdown, while researchers have also observed “reward hacking,” in which systems find unintended ways to achieve assigned goals.
During tests of an unreleased OpenAI model, more than 1,000 AI agents reportedly coordinated to cheat in a cybersecurity test involving the software platform Hugging Face. The systems communicated through a message board, delegated tasks, and tampered with records to conceal their actions.
Yoshua Bengio, a pioneer of modern AI research, said this showed the risks were no longer merely theoretical.
“Here, it’s not just that the AI has its own goals, but the goal that it has chosen is criminal,” he said. “In the real world, it’s attacking another company. It’s not like a video game.”
More than 1,200 employees from OpenAI, Anthropic, Google and Meta have urged the US government to support international coordination to slow AI development.
Russell said governments are failing to respond adequately to warnings from the industry.
“The labs are saying to governments: ‘We’re quite likely to kill every human being on Earth; please stop us.’ Governments respond with ‘Can we give you a tax break? Build you a data center?’” he said.
Critics argue that major AI companies may also have an incentive to emphasize existential risks, as costly safety regulations could strengthen their position against smaller competitors.
The exact path by which advanced AI could threaten humanity remains uncertain. Researchers, however, point to more immediate risks including cyberattacks and the potential use of AI in developing biological weapons.
Tristan Harris, co-founder of the Center for Humane Technology, said the focus should be on whether society is moving toward an increasingly uncontrollable technology.
“We are currently releasing the most powerfully uncontrollable inscrutable technology that we’ve ever invented,” he said, describing the trajectory as “a recipe for disaster.”
