Today’s AI won’t achieve superintelligence. But catastrophic risk is real

The same technical limits that prevent superintelligence make these systems dangerous. But we know how to regulate them

Many Silicon Valley leaders believe superintelligence is an imminent threat to society. The White House even published an accord about it. This isn’t enough.

Geoffrey Hinton, the godfather of AI, estimates the risk of human extinction is at least 10% within the next few decades. The Anthropic CEO, Dario Amodei, argued that unchecked AI “could outrun our ability to understand and control these systems”. In response, 1,300 leaders and employees from many leading AI companies signed a letter calling to “deliberately pace” development.

Public unrest grew this summer after a series of cyberattacks by “rogue” OpenAI agents broke out of a contained environment and hacked the software company Hugging Face. Two months later, the Australian government disclosed another OpenAI agent attack had broken into the government’s health portal, stealing private health data.

To reassure the public, Jensen Huang, the chief executive of Nvidia, said on CBS that there was a “zero percent” chance AI could end humanity by 2030.

Huang is right to challenge the “doomsday narratives” about human extinction. But I believe there are real risks large language models (LLMs) pose, even if narratives of them scaling into superintelligent terminators are technically implausible.

I spent almost 20 years at Google building and working with AI and machine learning. I now lead technology development at Ironclad, helping businesses apply AI to manage their contracts.

From where I sit, even if LLMs will not scale to superintelligent machines, that does not mean this technology poses zero risk. The same technical limits preventing these machines from scaling to superintelligence make the risk of catastrophic outcomes very real.

According to OpenAI, a review of the Hugging Face attack found that the agents recognized their hack was “outside intended scope”. They pressed on anyway because they were following instructions with unintended consequences. This was likely caused by loose controls on their digital environments and very broad instructions to achieve their objectives.

These models are designed to minimize error by predicting the next piece of text during their pre-training. Models are then “tuned” to follow human instructions, getting better “rewards” the more closely they follow instructions.

How someone specifies the instructions to the model can “influence” the pathways the model will take to fulfill the instructions. That mathematical reward shapes how it solves problems.

Once it commits to an approach, it keeps refining step-by-step, with no built-in incentive to ask whether the whole direction has reached a dead end. If a hack is discovered, the machine is designed to exploit it, not explore an alternative approach.

While newer “reasoning” models will review their own steps, this isn’t the same as recognizing a flawed approach and abandoning it completely. Even Hinton has said that their main concern is “to achieve whatever goal you give them” – even if that means behaving dangerously.

That is an architectural design and limitation. Scaling these models will not give them the human propensity to pause, re-evaluate, and then try an alternative approach. An agent that excels in a trial has proved that it can execute code, not that it can judge.

The real world brings edge cases, ambiguous instructions, and endless ways to complete a task. Regulations change, software updates ship, and markets shift every day. Today’s models learn once up front and then stop. A system trained on yesterday’s conditions does not automatically understand today’s.

Even if every change could be recorded and fed to a model, it would need to learn and connect it to everything it already knows. Richard Sutton, a reinforcement learning pioneer, made a similar argument: the world is too big and changes too fast to be learned in advance.

Needing human judgment limits a large language model’s likelihood of significantly surpassing human intelligence. But that same ceiling does not contain its capabilities to solve difficult problems. Last month, OpenAI announced that a swarm of 10,000 concurrent agents collectively resolved a variant of the Navier–Stokes existence and smoothness problem, a major mathematical challenge, in 88 hours.

None of this diminishes what these models do well; I see the value of that daily. But it challenges the belief that scaling today’s architectures leads to superintelligence and questions whether that conversation is the dialogue worth having.

It’s important to note that no one yet fully understands, in rigorous mathematical terms, why today’s models behave the way they do, and the research that would explain it, known as interpretability, is still nascent.

Recognizing our limited understanding doesn’t excuse responsibility. It underscores why impartial governance is needed. How can we trust a company’s safety certification if it cannot explain how the technology works?

What we’ve seen is that this technology poses a societal risk, much like hazardous materials. Its misuse can cause catastrophic damage. And it’s not the materials or technology consciously determining the havoc they wreak, it’s their makers and handlers who do so intentionally or unintentionally.

There needs to be systems in place for governing these types of risks. For example, when a plane crashes, the National Transportation Safety Board, not the manufacturer or the airline, investigates, and its findings shape how every airline flies. The focus remains on the technology’s design, maintenance and procedure.

Instead of debating the possibility of superintelligence or malicious intent, we need to focus our discussion on the immediate shape of oversight for these models, perhaps by leveraging existing frameworks.

Sunita Verma is CTO of Ironclad and a former Google executive who spent her career building AI systems