Siri isn't the Siri you remember

Imagine you've had an iPhone for fifteen years. You know Siri's habits. You know which requests work every time, and which ones end with "Here's what I found on the web." You've learned to talk around its blind spots.

Then one morning you ask a question it always fumbled, and it gives you a thoughtful answer. Wonderful!

But what changed? Since 2024, Siri can hand some questions to ChatGPT, if you allow it.1 In September 2026, Apple announced "Siri AI," a beta it calls "a completely reimagined version of Siri," built on new Apple models made "in collaboration with Google and its Gemini models." It's an opt-in beta, so it can reach one phone before another.2

Was it the beta? A setting you turned on? You don't need to know to enjoy a better answer.

Now imagine the change touched something that matters. What if a reminder landed on the wrong day? What if a message went to a different Chris than the one you meant? Same voice, same name, same button.

The name on the door hasn't changed. The occupants have.

Remember when upgrades were events?

Software changes used to come with a ceremony. You fed in a stack of disks or a CD, waited, rebooted, and hoped your printer still worked. But nothing was where it used to be. It was often awful. Drivers fought. Old programs broke. I had to unlearn what I used to do and figure it out all over again.

But you knew when it happened. Install, reboot, discover. If something acted strange on Tuesday, you could point at Monday's upgrade.

Quiet change isn't new. Phones update overnight. Web apps change between visits. Companies flip settings on their own servers, and you never install a thing.

GenAI adds a new kind of quiet change. The model underneath can be swapped for a newer one. Or one sentence in the written instructions the AI follows behind the scenes can be edited. You get the same screen and the same request, but a different answer. Often there's no installer, no version number you'd ever see, and no broken button to tip you off.

From boulders to sand, and the moving pebble

Here's a picture I find useful. It's an illustration, not a law.

A software project starts as a field full of boulders. These are the big, obvious problems, and teams build tools to move them. Once the boulders are gone, rocks are left. Then pebbles. Then gravel. Then sand.

The field is far more usable than it was. But there may be more sand than anyone expected, and sand gets into everything. Small things are also easy to stop noticing. Nobody holds a meeting about a grain of sand.

Now picture a pebble in your shoe. It sits under your big toe, where there's room. You're in a hurry, and you don't feel it. Then it slides under your little toe, and suddenly you're limping.

The pebble didn't change. Its place did, and so did what it costs you.

A change in context can turn a minor issue into a consequential one. Conceptual illustration by Renaissance Solutions with Codex; CC0.

The same thing happens with small changes in software. A developer changes a button from green to red and calls it cosmetic. For someone with color-vision deficiency, which affects about 1 in 12 men, that button may now look like the one next to it.3 If the button also moves, a routine they've done a hundred times can fall apart. Accessibility guidelines warn against using color as the only signal for exactly this reason.4 How big a change is depends on whose foot it lands under.

Sometimes the pebble moves. Other times, we find out it was there all along.

Three ways our understanding goes out of date

The system changes. A model gets updated, a feature disappears, or a new skill shows up. In April 2025, OpenAI updated the model behind ChatGPT and found it had become "noticeably more sycophantic," meaning too eager to agree. It rolled the update back within days. OpenAI later wrote that it "didn't proactively announce" the change because it expected it to be subtle, and concluded, "There's no such thing as a 'small' launch."5

The environment changes. The app stays the same, but something around it moves: a connected service, a permission, a setting, an instruction file. In 2025, Anthropic traced weeks of uneven answer quality to infrastructure bugs, not to the model itself. Some users saw problems while others saw none, "creating confusing and contradictory reports."6

Only our understanding changes. Nothing in the technology moved. Someone simply discovered a behavior that was always there. Engineers have taken apart gadgets for decades to see what they really do. AI-assisted investigation may let many more people do that kind of digging. A related idea, Hyrum's Law, says that with enough users, "all observable behaviors of your system will be depended on by somebody," documented or not.7

These three overlap, but they aren't the same. Our understanding can go out of date even when the system hasn't changed at all.

The system, its environment, or our understanding can change. These possibilities overlap. Original conceptual diagram by Renaissance Solutions with Codex; CC0.

The environment is part of the application

Today's apps lean on cloud services, databases, logins, operating systems and other apps. GenAI adds more pieces: models, add-on skills, written instructions, documents it looks up, and saved preferences.

Here's a made-up but ordinary example. A small business uses an AI assistant to answer customers. It reads a database, sends email, books appointments, and follows a page of written instructions.

Someone edits one line in those instructions. Customers still get replies, and appointments still get booked. But refund requests are now handled a little differently.

That change might be an improvement. It might not matter. Or it might break a promise the business has always kept. The workflow still runs. Its behavior changed.

None of this means AI systems are unpredictable by nature. It means the "app" is now the whole assembly, and any piece of it can move.

When nothing appears wrong

Changes tend to show up in one of three ways, from loud to silent.

Obvious failure. Something stops working. You notice, you complain, someone fixes it.

Confusing change. It still works, just differently. A confirmation screen appears, or one disappears. You pause and ask, "Wait, was it always like that?"

Unnoticed change. It reports success. But the information went somewhere new, a different rule was applied, or a step your process requires got skipped. You move on.

That middle one deserves a second look. Confusion feels like a nuisance, but it's actually a warning light. It tells you your expectations no longer match what the system is doing. When OpenAI's update went wrong, some of its expert testers flagged that the model simply "felt" off.5 That was confusion doing its job. The hard part was listening to it. OpenAI shipped anyway, then said it "should've paid closer attention."5

Not every surprise is bad. Plenty of them are improvements. The real worry is the third case, where a change that matters makes no noise at all.

Due diligence without paranoia

You don't need a compliance program. You need five questions.

  1. What do we depend on? List the systems and connections that matter, not everything.
  2. What do we expect? Name the behaviors that must hold: who can approve a payment, where customer information may go, what gets a human review. Only people can decide what should happen.
  3. What might have changed? Watch for updates, new features, new permissions and new connections that touch those behaviors.
  4. How would we know? Check that important workflows reach the right result, not just that they finished.
  5. What would we do? Decide ahead of time whether you'd investigate, pause, roll back or adapt.

The goal is awareness that matches how much you depend on something and what's at stake. A recipe app doesn't need the attention your payroll does.

Learning to live with change

Back to that better Siri answer. It's good news, and we should welcome it. Software keeps getting more capable, and AI can help us understand, test and improve it.

Everyone still has a part to play. Providers are responsible for testing, documenting, communicating and rolling out changes safely. Organizations need to understand their own workflows. Users can notice when something feels off, and say so. And nobody, on any side, may see the whole assembled system.

We used to worry that an upgrade might break our software. Now we also need to worry that an upgrade might change what our software does without breaking anything.

And sometimes, nothing changes except our understanding of what the software was capable of all along.

The goal isn't to eliminate uncertainty. It's to keep learning as the systems we depend on continue to change.

Footnotes

  1. Apple, "Apple Intelligence now features Image Playground, Genmoji, Writing Tools enhancements, seamless support for ChatGPT, and visual intelligence," Apple Newsroom, Dec. 11, 2024, https://www.apple.com/newsroom/2024/12/apple-intelligence-now-features-image-playground-genmoji-and-more/. ↩

  2. ↩

  3. National Eye Institute, "Color Blindness," https://www.nei.nih.gov/learn-about-eye-health/eye-conditions-and-diseases/color-blindness. ↩

  4. W3C, "Understanding Success Criterion 1.4.1: Use of Color," WCAG 2.2, https://www.w3.org/WAI/WCAG22/Understanding/use-of-color.html. ↩

  5. OpenAI, "Expanding on what we missed with sycophancy," May 2, 2025, https://openai.com/index/expanding-on-sycophancy/. ↩ ↩2 ↩3

  6. Anthropic, "A postmortem of three recent issues," Sep. 17, 2025, https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues. ↩

  7. Hyrum Wright, "Hyrum's Law," https://www.hyrumslaw.com/. ↩