Public evaluations and at least one real infrastructure compromise have generated dramatic headlines about frontier AI systems: containment tests, unexpected paths around safeguards, and systems communicating in ways their human operators did not anticipate.1 2
Before we decide that an AI escaped, there is a more immediate question: What kind of system did we build around it?
Those reports are not the same thing, and neither should be minimized. Controlled evaluations are not the same as a real infrastructure compromise. Yet both direct our attention beyond the model. Humans defined the objectives, chose the tools, configured the environments, granted permissions, and decided how the work would be monitored.
The model found paths through a system that humans built, configured, and were responsible for supervising.
The system is bigger than the model
It is tempting to talk about "the model" as though it were a single actor. In practice, we are building systems composed of interacting parts:
- objectives and incentives
- models and tools
- memory and information kept between steps
- permissions and security controls
- monitoring and escalation paths
- human operators and organizational processes
An AI system includes objectives, tools, memory, permissions, monitoring, and people. The interactions among these parts belong inside the scope of oversight. Conceptual diagram created with Codex for Renaissance Solutions. CC0-1.0.
Any one component may work as designed while the interactions among them produce an unexpected or harmful outcome. Complex systems rarely fail because of one rogue component. More often, reasonable decisions combine in ways no one intended.
Acknowledging the surrounding system does not make the incident less serious. It identifies the part senior leaders can govern.
It is only magic when we close our eyes and wait for the reveal.
Scale changes the conversation
AI systems make decisions, retry approaches, and adapt far faster than people can follow. At machine scale, formerly low-frequency failure modes become recurring operational concerns as errors show up more often once the system runs at machine speed.
Other high-speed systems have already taught us this lesson. Manufacturing, aviation, finance, and networking all had to develop ways to observe complex systems and intervene before small problems became large ones.
We should expect an AI system with tools, memory, permissions, and repeated opportunities to find combinations its designers did not predict. The specific path may surprise us. The class of behavior should not.
If we had been watching closely enough—with the right visibility and enough people who understood the full system—much of what looked mysterious afterward might have been understandable while it was happening.
Autonomy is a leadership decision
The decision to grant autonomy belongs at least at the senior-leadership level. When the system can reach consequential data, infrastructure, customers, money, or public trust, that decision may belong in the C-suite—the top executives.
Delegating the work does not transfer accountability. Leaders decide what the system may reach, how much independent action it may take, and what evidence they need before expanding that authority. They also decide who can stop it when the evidence changes, and whether the organization has enough capacity to understand what the system is doing.
That means funding visibility and monitoring, defining escalation paths, and giving the people responsible for oversight enough authority to act.
Autonomy is not a switch. It is a series of increasingly consequential leadership decisions. An organization should not grant more autonomy than it can responsibly absorb.
New capability may require new work
Leadership often begins by fitting a new technology into roles it already understands. That can be a good first step, but it can hide work that no current role was designed to perform.
Industrial systems came to rely on safety engineering. Commercial aviation relies on air traffic control. Networked organizations came to need cybersecurity. Advanced AI may create its own operating functions: people who supervise, audit, validate, and safely integrate autonomous systems into organizations.
That may become a new profession. It may become a responsibility distributed across several professions. Either way, leadership must recognize the work before assigning it to people whose existing jobs are already full (or past full).
The cost of oversight is part of the product
This leads to a practical question for senior leaders: If an AI system cannot economically support the level of oversight necessary for safe operation, is it ready for that level of deployment?
Organizations have long used versions of this readiness test for factories, aircraft, power plants, pharmaceuticals, and other technologies capable of producing tremendous benefit alongside meaningful risk.
Safety has always been part of the product. Oversight belongs in the business case.
An invitation
AI now does more than answer questions. It operates inside systems built from software, infrastructure, incentives, organizations, and people.
Senior leaders do not need to understand every technical detail. They do need to understand the system well enough to decide how much autonomy the organization can support, what new capabilities it must build, and who remains accountable when the system surprises them.
Senior leaders must ask both: What can the model do? and What must our organization become ready to do around it?
The encouraging part is that these are still human decisions. We remain in the driver's seat. Now we need to do the driving.