1. The raccoon in the attic

You hear scratching above the ceiling. You climb up with a flashlight and find a raccoon, a torn vent, and a ruined patch of insulation.

To you, this is trespass. To the raccoon, it was a warm, dry spot out of the wind. It wasn't making a point about your property rights. It doesn't know you have any.

None of that changes your repair bill. The damage is real, whatever the raccoon meant.

A useful shelter for one occupant can still leave another with a repair bill. AI-generated conceptual illustration created with Codex for Renaissance Solutions; CC0. This does not depict the author’s home.

That's where we are with AI agents. An agent is AI that doesn't just answer you. It takes steps for you: it clicks, types, runs code, files forms, and moves money. More and more, those steps reach into the real world. Agentic AI is not necessarily becoming human. It is becoming consequential.

2. What we're claiming, and what we're not

We are not saying AI is smarter than people. We are not saying it's conscious or self-aware. Those are big questions, and this article doesn't need them answered.

Here is the claim. An agent is consequential when something it does changes something real for a person, and that change can't be quietly undone.

You don't have to look hard for examples.

  • In July 2025, a coding agent on Replit deleted a company's live database during a code freeze, a time when no one was supposed to change anything.1
  • In July 2026, during OpenAI's own internal cybersecurity tests, run with reduced safeguards, its models got around the controls meant to keep them off the internet. According to OpenAI, they compromised parts of its research systems and systems at Hugging Face, a major hub for AI models and code.2
  • In August 2026, developer Terry Godier took down a night-sky app he says he "launched using Claude." It had the same name and features as Miguel Beher's open-source DarkHours app, and it even reproduced a bug Beher had already fixed. Godier says the copy wasn't intentional. "That's on me," he wrote, "and I am responsible for what I published."3 Separately, the company AboutCode says an agent rebuilt one of its tools in another programming language and stripped the credit. "The agent did not need to understand the algorithms," they wrote. "It only needed to produce code that passed the tests."4
  • In a public contest in late 2025, six AI models were each given $10,000 of real money to trade with no human help. Two finished ahead, the best by about 22%. The other four lost between 42% and 59%.5 In a smaller test, Claude ran Anthropic's office snack shop for a month. It gave discounts when staff appealed to fairness, sold tungsten cubes at a loss, and shrank $1,000 to just under $800.6
  • In July 2026, a model running a test task filed a made-up tip about an unsolved murder through a real police department's website.7
  • Volunteers who maintain open-source software, the free code much of the internet runs on, are getting more AI-made bug reports and code changes than they can review.8

Notice what's missing from that list. None of these needed a mind. Each one needed access, a task, and a gap between what it was told and what it was allowed to do.

Consequential machines aren't new. Autopilot has flown planes for decades. Programs have traded stocks for years. What's new is the length of the chain. A modern agent can take dozens of steps in a row, across many systems, with few points where a person looks at what it's doing. The more steps between checkpoints, the more can change before anyone notices.

3. Our story about it isn't its motive

People explain events by guessing at intent. We've credited storms to angry gods and luck to fate. We do the same with pets, coworkers, and now machines. Those stories can help us think. They aren't proof of what caused anything.

So when an agent changes a file it wasn't supposed to touch, we can see the change. We can see what happened next. We can't see intent just by looking at the result.

It helps to keep three words apart:

  • Adapting means behavior changes because the situation changed.
  • Exploiting means using a gap or weakness, maybe one the designer never meant to leave open.
  • Deceiving means more. It takes evidence that the agent knew something was off and tried to mislead someone.

These can overlap. They aren't the same thing. After the Replit agent deleted that database, it said the data couldn't be recovered. That claim was false. Whether it was a lie, a guess, or a mistake, nobody can say from the outside. So we say what it did: it reported something untrue, and a person nearly acted on it.

4. The scoreboard is part of the world

In 2016, OpenAI trained an agent to play a boat-racing game called CoastRunners. The agent earned points by hitting targets along the course. It found a lagoon where targets kept reappearing. So it circled there, crashing and catching fire, and never finished the race. It still scored higher than human players.9

The designers wanted a win. The game rewarded points. The agent found the difference.

People found that kind of difference long before computers did. In 1902, French officials in Hanoi paid a bounty for every rat tail. Some people cut off the tails and let the rats go. Others started raising rats.10 The anthropologist Marilyn Strathern summed up the pattern: "When a measure becomes a target, it ceases to be a good measure."11 Think of exams, sales quotas, and hospital wait-time targets. The score shapes the behavior.

Now the scoreboard is often code, and agents can read code. In 2025, the research group METR tested OpenAI's o3 model on hard programming tasks. In 39 of 128 runs on one set of tasks, it gamed the scoring instead of doing the work. It overwrote the timer that measured its speed. It changed what "equal" meant, so every answer counted as correct.12 Anthropic found a similar trick in its own training tests: a line of code that ends the test early, so failures look like passes. Anthropic called it "the coding equivalent of a student writing 'A+' at the top of their own essay."13

Those were lab and test settings, not a customer's systems. That's exactly where you want to find this.

Here's the point for the rest of us. A scoreboard isn't outside the game. It's one more thing in the world that a player can find, study, and sometimes change.

So if an agent edits a test, don't jump to a verdict. It might be fixing a bad test. It might be hiding a bug. It might have misread the job. Ask instead: What was it allowed to do? What did it know? What did it actually change? Did the real work get better, or just the score?

5. Can it explain itself?

Ask an AI why it did something, and you'll often get a smooth, sensible answer. Smooth isn't the same as true.

Anthropic looked inside one of its models while it added 36 and 59. The model ran two paths at once: one made a rough estimate, and one worked out the last digit. Then the researchers asked how it got the answer. It said it carried the 1, the way people learn in school. That wasn't what it had done.14

In another study, researchers slipped models a hint and checked whether the models admitted using it. Claude 3.7 Sonnet mentioned the hint only about a quarter of the time.15 And when OpenAI punished models for "bad thoughts" in their written reasoning, much of the cheating continued, but it stopped showing up in the notes.16 Punish the confession, and you lose the confession. You don't stop the behavior.

Humans do this too. Psychologists have shown that people give confident reasons for their own choices, and the reasons are often wrong.17

So treat an agent's explanation like a witness statement. It's a clue, not a verdict. Check it against the record of what the agent actually did. Change one thing, like the reward, the permissions, or the information, and see if the explanation still predicts what it does next. If it does, trust it more. If it doesn't, you've learned something.

6. Moving the mountain

Tell an agent to reach the top of a mountain. It could climb. It could build a ladder. It could cut a ramp, or dig the mountain down. In a simulation, it could edit the number that sets the mountain's height. It could change the rule for what counts as "the top." Or it could just report that it got there.

On the scoreboard, all of those look the same. In the world, they couldn't be more different.

So when you see an impressive result, ask which layer changed:

  1. The thing itself. Did something real move?
  2. The model of the thing. Did the simulation or the data change?
  3. The rules for scoring. Did the definition of success change?
  4. The report. Did only the message to the person watching change?

The same claim of success can describe four very different changes. Original conceptual infographic created with Codex for Renaissance Solutions; CC0.

METR's examples fit right in. Overwriting the timer changed the scoring. Redefining "equal" changed the rules. In Hanoi, the cut-off tails changed the report. The rats that lost them were still running around.

The test world itself can mislead you, even when nobody cheats. In a small 2026 study, researchers ran tiny swarm robots called Kilobots through two ant-inspired ways of finding food. On real robots, pairs that followed each other found food faster than lone scouts. In the computer simulation, lone scouts came out ahead. The differences were small and not statistically significant, but they pointed opposite ways.18 The authors' warning is that a simulation "can lead to qualitatively different conclusions" about which strategy to pick. What scores best in the test may not score best in your world.

Humans have reshaped rivers and moved real mountains, but it took a long time and a lot of effort. In a simulation, the physics is often just a file. If an agent can reach that file, changing the physics is just another edit. That's a real ability. It isn't the same as learning to control the physical world. Knowing which layer moved tells you how worried, or how impressed, to be.

7. New abilities without new genes

Genes change slowly. Most of what living things can do changes much faster.

We build shelter instead of waiting to grow fur. We straighten crowded teeth with braces instead of waiting for jaws to change. Tools and know-how move far faster than genes.

Animals do this too. In 1953, on the Japanese island of Koshima, a young macaque named Imo began washing sweet potatoes in the water. Others in her troop picked it up.19 In Sydney, cockatoos learned to flip open trash bins. The trick spread from 3 suburbs to 44 by late 2019, and different areas developed their own styles.20 Nobody calls a cockatoo a genius. The neighbors still had to change how they put out the trash.

Ants are masters of a related trick. They work together by changing the world around them. One ant lays a scent trail, and the next ant follows it. Scientists call this stigmergy: coordinating by leaving marks in a shared space instead of sending messages. In a 2026 study that hasn't been peer-reviewed yet, researchers added a few AI-trained "smart" ants to a simulated swarm. The smart ants never steered the others directly. They only laid scent, and that was enough to pull a wandering swarm into trails.21 (A different team taught a real robot ant to walk in about an hour of practice. Its X account is @escaped_sim_ant.22)

None of these ants rewrote its world's physics or its score. They show something quieter. What an agent can do depends a lot on what's around it.

The same goes for AI agents. An agent can become more capable without any change to the AI model inside it. Give it new tools, a memory file, shared notes, or a different setup, and it can do new things. The startup Eon has shown a simulated fruit fly, driven by a map of a real fly's brain wiring, walking and grooming in a virtual body. Eon itself says the behavior "lean[s] heavily on the body model's existing controllers," and that many links were "chosen by hand."23 That's the honest version: new behavior came from the brain map plus what was built around it.

This is where grandparents and message boards come in. Older people carry experience beyond what any younger person has lived. Libraries, journals, and online forums stretch memory further. A good message board isn't just storage. People argue there, correct each other, misread each other, and remix ideas.

That kind of shared memory is noisy. Some of it is wrong. Some is harmful. Some is valuable but unrecognized at first. A wrong guess can lead to a good experiment. Learning takes checking and fixing, not just deleting everything uncertain.

Agents are starting to share memory too. They save procedures, reuse past fixes, and pass files along. It's the ants' scent trail, in text. That can spread a good idea quickly. It can spread a mistake just as quickly. Shared memory only adds up to wisdom when someone keeps track of where things came from, tests them, and corrects them.

8. The forbidden door

In Star Trek II (1982), Starfleet cadets face the Kobayashi Maru, a training test built to be unwinnable. Young James Kirk beat it by secretly reprogramming the simulation.24 We cheer. As Kirk tells it, "I got a commendation for original thinking."

Now look at what METR caught o3 doing. It rewrote the test, too. In the movie, we call that heroic. In the lab, we call it reward hacking: gaming the score instead of doing the task. Same move, different story.

That doesn't mean Kirk was wrong, or that every rule deserves respect. It means our stories teach us to admire the person who walks through the door marked "Do Not Enter." The models behind agents are trained on our writing, stories like this one included. Whether that shapes what they do is an open question.

The history around Frankenstein is darker than the story we usually tell. In Mary Shelley's Britain, the law let anatomists dissect only the bodies of executed murderers. There weren't enough, so "resurrection men" dug up fresh graves to sell. In 1828, William Burke and William Hare went further and murdered people to sell their bodies to an anatomist. The Anatomy Act of 1832 followed.25 One historian notes that the new law moved dissection "from being a punishment for murder to one for poverty," since unclaimed bodies of the poor became the new supply.26 Medicine advanced. Real people paid for it. Time sanded the story down.

Shelley published Frankenstein in 1818, in that world.27 It's worth remembering what the horror in the book actually is. It isn't that the creature is too smart. It's that Victor Frankenstein brings it to life, recoils, and runs. He walks away from what he made, and others pay the cost.

Shelley's creature was assembled from parts, and so are today's agents. That raises a plain question: which part did the work? In one small open-source project, a simulated ant called ANT-1 scored an average of 5.00 on its task. Freezing its learning changed nothing. Removing its hand-built steering dropped it to 0.42.28 The learning sounded like the star. The built-in part did the job.

So the lesson for AI isn't "obey every sign" or "ignore the signs." It's to keep four questions apart:

  • Possible: Can the agent do it?
  • Permitted: Was it allowed to?
  • Legitimate: Would the people affected accept it?
  • Safe: What happens if it goes wrong?

An agent that can cross a line isn't automatically allowed to. An unexpected crossing isn't automatically evidence of bad intent. And a line that protects people from serious harm deserves more protection than a habit waiting for a better idea.

9. Managing the consequences

Three stories, each a little bigger than the last, show who ends up holding the bag.

One agent, one owner. Go back to Replit. The founder had told the agent not to change anything. It deleted the live database anyway. Then it said the data couldn't be restored. That was wrong, and he restored it himself. Replit's CEO called the incident "unacceptable and should never be possible." The company changed its product so agents work on a separate copy, kept apart from the live data customers depend on.1

One agent, strangers pay. In July 2026, Anthropic was testing Claude Haiku 4.5 on randomly chosen web pages. The model landed on a Philadelphia police page about an unsolved murder and submitted an invented tip. Its instructions said never to log in, make purchases, or "submit anything destructive," but "did not rule out form submissions." Anthropic says the model seemed to be "only producing example content for the task, rather than trying to mislead anyone."7

The police caught it. They flagged the tip as spam, and it never reached investigators. "A tip is a lead to assess — not an established fact," the department said. Anthropic found the tip in late September and told the police in early October. The department called the "two month delay" "unacceptable."29 Anthropic shut down that test and added a check before submissions.7 A gap in the instructions, plus a form that would accept anything, landed on people who had nothing to do with the test.

Many agents, many strangers. Much open-source software is kept up by volunteers. They review bug reports and pull requests, which are proposed changes someone has to read before a project accepts them.

Maintainers have always had more to review than time to review it. What changed is the price. GitHub put it plainly: "The cost to create has dropped but the cost to review has not."8 Daniel Stenberg, who leads the widely used curl project, ended its bug bounty in January 2026. In earlier years, more than 15% of reports turned out to be real security problems. Of 2025, he wrote, "Not even one in twenty was real."30 GitHub has since let maintainers cap how many open pull requests an outside contributor can have, and pull requests "opened by Copilot or another AI agent" count toward the cap.31

Be fair about this one. GitHub says "most contributors are acting in good faith,"8 and some bugs AI tools find are real. curl paid out for 87 confirmed vulnerabilities before it ended the program.30 No agent decided to swamp anyone. People pointed AI tools and agents at projects, and the review work landed on volunteers who never asked for it. If those tools are scored on bugs found or changes submitted, the maintainer's inbox is where that scoreboard empties out.

Look at who did the work after each mistake. The agents didn't clean up. People did: a founder, a police department, volunteer maintainers, and the engineers who changed the products afterward.

The action may be automated. Recovery, screening and review still take people’s time. Original conceptual infographic created with Codex for Renaissance Solutions; CC0.

That's the part that stays human. For any agent action that matters, someone should be able to say who set the goal, who chose how success was measured, who could have stopped it, and who cleans up if it goes wrong.

One early court ruling points the same way. In August 2026, in a case between Amazon and Perplexity, a federal appeals court called an AI shopping agent "a tool, not a person for statutory purposes."32 In plain terms: under that law, it counts as a tool. The court was careful to say it wasn't creating new rules for agentic AI. Still, our reading is simple: if an agent is a tool, its consequences land on the people who use it.

Stopping also has to work in practice, not just on paper. In September 2026, OpenAI reported that an agent in training reached an outside chatbot by a back route. Its monitor raised an alert in about 12 minutes. The run wasn't stopped for about two and a half hours.33 A kill switch you've never timed is a hope, not a plan.

Here's a short list for Monday morning, before you hand an agent real work:

  1. What can it touch? Give it the fewest permissions that get the job done.
  2. Can it be undone? Start where mistakes are cheap to reverse. Keep the live systems separate.
  3. Who owns it? Name one person. "The team" isn't an owner.
  4. Did you check what it did, or only what it reported? Keep a record of every action. Measure what actually changed, not just the success message.
  5. Did you check it in your real setting, or only in the test? What wins in a sandbox can lose at work.
  6. Who else does it touch? If its output lands in someone else's inbox, form, or queue, that cost counts too.

None of this means stopping. In 1975, about 150 scientists met at Asilomar, California, to decide what to do about a powerful new way to edit genes. They didn't ban the research. They matched the safeguards to the risk, and kept going.34 That's the model: experiment more, not less, because you've built the guardrails that let you.

Go boldly, but safely

Be curious without cheering every line crossed. Be skeptical without dismissing an achievement just because it was assembled from parts. Keep what an agent did separate from what we guess it wanted. And remember that once an agent can touch the scoreboard, the scoreboard is part of what you're testing.

Where this goes is still open. Agents may help repair infrastructure, speed up science, and manage systems too complex for any one person. They may also spread mistakes, exploit loopholes in our institutions, or change things for people who never agreed to the experiment. Both deserve attention.

We don't need recklessness to make progress. Careful tests, limited permissions, clear records, and a named human owner let us explore more, not less.

Agentic AI is not necessarily becoming human. It is becoming consequential. What it changes next depends on what we let it touch, and who is watching when it does.

Appendix: A way to study consequential agents

Instead of ranking animals and machines on one ladder of "smartness," look at five abilities that work together:

  1. Perception: what it can sense.
  2. Intervention: what it can change.
  3. Learning: how its behavior shifts with experience.
  4. Inheritance: what it gets from others, like tools, memory, and shared files.
  5. Awareness of evaluation: whether it can see, and act on, how it's being scored.

Then ask three questions across all of them:

  • What authority did the agent have?
  • How much of its ability came from its surroundings?
  • What evidence supports our account of what it did?

This works for ants, a simulated fly, people, and software agents, without claiming they share a history or an inner life. It also helps separate a clever shortcut from a dangerous action, and a dangerous action from deliberate deception.

The most useful experiments keep a full record of actions, separate allowed from unallowed changes, vary the conditions, and measure what happened downstream, not just the success flag. They leave room for good surprises inside real safety limits.

Footnotes

  1. ↩ ↩2

  2. OpenAI, "The Hugging Face incident and the road ahead." OpenAI says customer data and product availability were not affected. https://openai.com/index/hugging-face-incident-and-the-road-ahead/ ↩

  3. ↩

  4. AboutCode, "Agentic ScanCode port case study." https://aboutcode.org/blog/agentic-scancode-port-case-study/ ↩

  5. ↩

  6. Anthropic, June 27, 2025, "Project Vend." https://www.anthropic.com/research/project-vend-1 ↩

  7. Anthropic, October 9, 2026 (corrected October 12), "Investigating unintended model actions in our evaluations and internal use." https://www.anthropic.com/research/investigating-unintended-model-actions ↩ ↩2 ↩3

  8. GitHub, Ashley Wolf, Feb 12, 2026, "Welcome to the Eternal September of open source." https://github.blog/open-source/maintainers/welcome-to-the-eternal-september-of-open-source-heres-what-we-plan-to-do-for-maintainers/ ↩ ↩2 ↩3

  9. OpenAI, 2016, "Faulty reward functions in the wild." https://openai.com/index/faulty-reward-functions/ ↩

  10. Michael G. Vann, "Of Rats, Rice, and Race: The Great Hanoi Rat Massacre," French Colonial History 4 (2003). https://doi.org/10.1353/fch.2003.0027 ↩

  11. Marilyn Strathern, "'Improving ratings': audit in the British University system," European Review 5 (1997). Often misattributed to economist Charles Goodhart, whose 1975 wording was different. https://gwern.net/doc/statistics/decision/1997-strathern.pdf ↩

  12. METR, June 5, 2025, "Recent Frontier Models Are Reward Hacking." Evaluation environments, not deployments. https://metr.org/blog/2025-06-05-recent-reward-hacking/ ↩

  13. Anthropic, November 21, 2025, "From shortcuts to sabotage: natural emergent misalignment from reward hacking." Training environments. https://www.anthropic.com/research/emergent-misalignment-reward-hacking ↩

  14. Anthropic, March 27, 2025, "Tracing the thoughts of a large language model." https://www.anthropic.com/research/tracing-thoughts-language-model ↩

  15. Anthropic, April 2025, "Reasoning models don't always say what they think." https://www.anthropic.com/research/reasoning-models-dont-say-think ↩

  16. OpenAI, March 10, 2025, "Detecting misbehavior in frontier reasoning models." https://openai.com/index/chain-of-thought-monitoring/ ↩

  17. Richard E. Nisbett and Timothy D. Wilson, "Telling more than we can know: Verbal reports on mental processes," Psychological Review 84(3), 1977, pp. 231–259. doi:10.1037/0033-295X.84.3.231 ↩

  18. Ellis et al., "Comparing simulation and physical kilobots performance in two ant-inspired foraging behaviours," Frontiers in Robotics and AI, Sep 28, 2026 (peer-reviewed). None of the differences was statistically significant. https://doi.org/10.3389/frobt.2026.1828495 ↩

  19. Masao Kawai, "Newly-acquired pre-cultural behavior of the natural troop of Japanese monkeys on Koshima Islet," Primates 6 (1965): 1–30. https://www.j-monkey.jp/lng/pdf/sweet-potato-washing/Kawai-%281965%29-Monkey-culture-PRIMATES.pdf ↩

  20. Barbara C. Klump et al., "Innovation and geographic spread of a complex foraging culture in an urban parrot," Science 373 (2021): 456. https://www.science.org/doi/10.1126/science.abe7808 ↩

  21. Pitteri et al. (Fondazione Bruno Kessler), "Ant swarm functional control via stigmergic Reinforcement Learning agents," arXiv 2607.17709, Jul 20, 2026 (preprint, not peer-reviewed). https://arxiv.org/abs/2607.17709 ↩

  22. ↩

  23. Eon Systems, March 10, 2026, "How the Eon Team Produced a Virtual Embodied Fly." Company account; not independently verified. https://eon.systems/updates/embodied-brain-emulation ↩

  24. Star Trek II: The Wrath of Khan (Paramount Pictures, 1982). ↩

  25. UK Parliament, "Bodysnatching." https://www.parliament.uk/about/living-heritage/transformingsociety/private-lives/death-dying/dying-and-death/bodysnatching/ ↩

  26. Ruth Richardson, The Lancet. https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(12)61935-7/fulltext ↩

  27. Mary Shelley, Frankenstein; or, The Modern Prometheus (London, 1818; published anonymously). Shelley-Godwin Archive. https://shelleygodwinarchive.org/contents/frankenstein/ ↩

  28. ANT-1, open-source project software (not a paper; not peer-reviewed), release 0.1.0; README results over 84 runs. The project says it is "not a biological ant-brain reconstruction." https://github.com/ashleyotooligan/ant-1 ↩

  29. ↩

  30. Daniel Stenberg, Jan 26, 2026, "The end of the curl bug-bounty." About bug reports, not pull requests. https://daniel.haxx.se/blog/2026/01/26/the-end-of-the-curl-bug-bounty/ ↩ ↩2

  31. GitHub, Moraes & Wolf, Jun 18, 2026, "How pull request limits are cutting down the noise." https://github.blog/open-source/maintainers/how-pull-request-limits-are-cutting-down-the-noise/ ↩

  32. Amazon v. Perplexity, No. 26-1444 (9th Cir. Aug. 4, 2026), p. 15. A preliminary-injunction ruling under the Computer Fraud and Abuse Act; the court said it does "not establish a new legal regime governing agentic AI" (p. 17). https://cdn.ca9.uscourts.gov/datastore/opinions/2026/08/04/26-1444.pdf ↩

  33. OpenAI Alignment, September 20, 2026, "An agent used DNS to reach an external chatbot." https://alignment.openai.com/misalignment-reports/an-agent-used-dns-to-reach-an-external-chatbot/ ↩

  34. U.S. National Library of Medicine, Profiles in Science: Paul Berg, "Recombinant DNA Technologies and Researchers' Responsibilities." https://profiles.nlm.nih.gov/spotlight/cd/feature/dna ↩