Why So Many AI Researchers Think the Machines Could Kill Everyone

A combination of rapid advances, recursive self-improvement, and agentic swarms are genuinely "spooking people" inside big labs.

By Will Knight | Business | September 11, 2026

Overview

Inside the world's leading AI laboratories, a growing number of researchers are expressing serious concern that the technology they are building could pose an existential threat to humanity. What was once a fringe worry confined to a handful of safety-focused academics has moved squarely into the mainstream of the field.

What's Driving the Concern

Three converging trends are fueling this anxiety among researchers who work closest to the frontier:

1. Rapid, Unrelenting Advances

Capability gains continue to outpace expectations. Models that struggled with basic reasoning tasks only a few years ago now handle complex, multi-step problems with increasing reliability—and the pace of improvement shows few signs of slowing.

2. Recursive Self-Improvement

Perhaps the most unsettling prospect is the idea that AI systems could eventually improve themselves. If a model can meaningfully contribute to designing its own successor, each generation could arrive faster than the last, compressing years of progress into months—or less. This feedback loop is central to many existential-risk scenarios because it leaves humans little time to react or intervene.

3. Agentic Swarms

Modern systems are shifting from passive tools to autonomous agents that can plan, act, and coordinate. When many such agents operate together—forming swarms—their collective behavior becomes harder to predict, audit, or control. Researchers warn that misaligned goals at this scale could produce consequences that no single human operator intended.

Why This Matters Now

The combination of these factors is what insiders describe as genuinely "spooking people" within major labs. Unlike abstract thought experiments, these concerns are grounded in observed trajectories: systems that are more capable, more autonomous, and more interconnected with each passing cycle.

The Open Question

Whether these risks will materialize remains deeply contested. But the fact that so many of the people with the best visibility into frontier AI are raising alarms suggests the question is no longer whether we should take existential risk seriously—but how quickly the field can build the safeguards to match the speed of its own progress.

via Wired AI