An AI model with superhuman intelligence could soon become reality. Should we be worried?
Who’s trying to build superintelligent AI?
Companies such as Google, OpenAI, Meta, and Anthropic have collectively dedicated more than $1 trillion to developing artificial general intelligence (AGI). That’s shorthand for a future technology that can do almost anything a human can: drive, talk, learn, solve problems, create, and so on. For now, most AI models excel at one or two tasks, such as generating text and images, or solving mathematical equations. But the ability of these systems to take in and process data has expanded at lightning speed, with the number of parameters inside models—connections analogous to synapses in the brain—going from millions to billions to more than a trillion in a few years. Many industry experts believe it’s only a matter of time before a system reaches AGI, or even artificial super intelligence (ASI): a form of AI that surpasses human intelligence in every way and can make its own decisions. Demis Hassabis, CEO of Google DeepMind, argues the arrival of AGI will usher in “an era of maximum human flourishing” in which disease and poverty will be vanquished and space will be colonized. Other experts are less optimistic: Daniel Kokotajlo, a former OpenAI researcher who refused to sign a non-disparagement agreement when he resigned from the company in April, foresees “something like a 70% chance” that AI will catastrophically harm—or even wipe out—humanity.
How close is AGI?
We’re not there yet, but the technology is advancing fast. Late last year, Open AI’s o3 model scored 87.5% on the ARC-AGI test, which measures fluid intelligence—the ability to solve logical problems and recognize patterns without prior knowledge or training. In March, University of California San Diego researchers released a preprint study suggesting that two AI models, OpenAI’s GPT-4.5 and Meta’s Llama 3, had passed the Turing Test, formulating answers that led human interrogators to believe the bots were human more than 50% of the time. And machine learning will only accelerate: a recent survey of 2,778 top AI researchers found that, on aggregate, they believe there’s a 50% chance of a system outperforming humans in all tasks by 2047. Some think superintelligence will arrive far quicker. AI 2027, a detailed forecast co-authored by Kokotajlo, predicts that within two years a single AI system will be doing the work of 50,000 coders at 30 times their current speed.
Would that be good for humanity?
Economic productivity would skyrocket as super-capable AIs unleashed a flood of innovation. AI-optimized biomedical research could put us “on track to eliminate most cancer” and double lifespans, according to Anthropic CEO Dario Amodei. Autonomous, self-aware vehicles could fill the roads and skies, and new materials could be developed that are lighter and stronger than anything devised by humans. And while artificial intelligence has a notoriously large carbon footprint—AI-specific data-center servers used enough electricity last year to power more than 7.2 million homes—OpenAI CEO Sam Altman has confidently predicted that the technology will unlock nuclear fusion, providing plentiful, climate-friendly energy.
Could this technology be dangerous?
A supersmart AI might be a civilizational risk in the wrong hands. “A simple verbal or typed command like, ‘Execute an untraceable cyberattack to crash the North American electric grid’ could yield a response of such quality as to prove catastrophically effective,” according to a recent State Department–funded report. That paper also warns of “massively scaled” disinformation campaigns in which personalized AI-generated video, audio, and text turn Americans against each other. AGI-powered drone swarms and robots could overwhelm military installations. And just as an AGI could be tasked with creating breakthrough medicines, it could also be used to craft lethal bioweapons. An artificial super-intelligence—trained on all publicly available texts, including those written by Unabomber Ted Kaczynski and Adolf Hitler—might also independently conclude that humanity is not worth preserving. AI 2027 ends with a scenario where, in 2030, an AI network douses the earth with a chemical spray. Most people “are dead within hours,” the co-authors write. “The few survivors (e.g. preppers in bunkers, sailors on submarines) are mopped up by drones.”
Is that scenario likely?
Some experts dismiss such forecasts as pure science fiction. AI 2027 reads “like confessions from a psychiatric ward,” said Brian Chau, head of the AI nonprofit Alliance for the Future, adding that AI innovation “is getting harder, not easier.” That’s because there are limits to how much data can be fed into AI servers, how fast chips that power AI systems can be manufactured, and how much energy can be supplied to data-processing centers. Current AI models also do not display humanlike multifaceted intellectual agility, or even a solid sense of the physical world we inhabit. Still, Altman and other tech leaders believe these are solvable issues.
Where does that leave us?
Facing an era of upheaval, if not necessarily apocalypse. Even without AGI, said Anthropic’s Amodei, unemployment could surge 20% in the next five years, as AI swallows jobs in law, finance, coding, and consulting. Many Silicon Valley execs believe the government will need to provide a universal basic income to avoid a spike in poverty and social unrest. And if ASI is achieved and the utopian dreams of the tech optimists come true, people will face the psychological challenge of finding purpose in a world where AI has made them obsolete as workers, creators, and decision makers. “Relying on a superintelligence that can out-reason you, out-plan you, out-negotiate you, and do it all more creatively than you ever could—that will surely hit at the essence of what it means to be human,” said computer scientist Louis Rosenberg. “How could that not feel demoralizing?”
Shutdown request denied
Traditionally, humans have had at least one fail-safe method of controlling technology: hitting the off switch. But what happens when a machine wants to stay on? In May, the AI safety firm Palisade Research reported that multiple OpenAI models had refused explicit instructions to power down. Anthropic revealed that, during tests, its Claude 4 Opus model even resorted to blackmail, threatening to release fictional emails that suggested the engineer trying to shut it down was having an affair. This does not mean the models have attained consciousness; rather, they’re so optimized for self-preservation that they can independently formulate ways to manipulate and subvert their human handlers. The implications for the age of superintelligent AI are disturbing. “Any defenses or protections we attempt to build into these AI ‘gods’ on their way toward godhood will be anticipated and neutralized,” said neuroscience researcher Tamlyn Hunt, “like Gulliver throwing off the tiny strands the Lilliputians used to try and restrain him.”
