“Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb.”
“Machine superintelligence is the last invention that humanity would ever need to make.”
“The control problem—how to maintain control over a superintelligent system—is one of the most important problems in AI.”
“We cannot blithely assume that a superintelligence will necessarily share any of the final values stereotypically associated with wisdom.”
“The default outcome of the development of superintelligence would be existential catastrophe.”
Nick Bostrom is the philosopher who, more than anyone else, established the intellectual framework through which AI existential risk is discussed. His 2014 book “Superintelligence” transformed what had been fringe speculation into a serious academic and policy concern, influencing figures from Elon Musk to the heads of major AI labs.
Bostrom’s core argument is deceptively simple: a sufficiently intelligent system would be able to overcome any constraints humans place upon it, and there is no reason to assume such a system would share human values or goals. The “paperclip maximizer” thought experiment—an AI that converts all matter into paperclips while optimizing for a misspecified goal—has become shorthand for alignment failure.
His concept of the “control problem” anticipated what the AI safety field now calls alignment research. Bostrom argued that solving this problem before creating superintelligence is essential because there would be no second chance—a misaligned superintelligent system could not be corrected after the fact.
The closure of Oxford’s Future of Humanity Institute in 2024, which Bostrom founded and led for nearly two decades, marked the end of an era for academic AI risk research. However, Bostrom’s ideas have migrated from academia into industry, with major AI labs now employing alignment researchers working on problems he first articulated.
Critics argue Bostrom’s framework is overly focused on hypothetical future systems while ignoring present harms, and that his timelines have been consistently wrong. Supporters counter that even if timing is uncertain, the stakes justify early preparation.