People still talk as if AI exposure equals “the AI products we bought.” But, we continue to learn that: you can no longer determine your AI exposure simply by inventorying the AI tools your organization intentionally uses.

We are increasingly participants in adaptive systems, not merely users of finished technology. AI can enter decisions through vendors, suppliers, software, search, news, analytics, summaries, risk models, customer systems, consultants, and other intermediaries — whether or not anyone “uses AI” on purpose.

That is why a new leadership assessment is worth watching. Not as a finished playbook. As an experiment that may teach the institution about the assessment itself.

Roosevelt Hall, home of the National War College at Fort McNair in Washington, DC. The Naval War College discussed below is in Newport, Rhode Island. Photograph: National Park Service, public domain in the United States.

What is reported about the Joint Warfighter Evaluation

Reported: The Pentagon is adding a Joint Warfighter Evaluation (JWE) that will supplement existing promotion processes for officers seeking their first star.1

Service implementations differ. Some rely heavily on remote or digital simulations. The Navy plans an in-person assessment at the Naval War College. Marine implementation reportedly includes an AI-enabled assessment platform.

The Pentagon has not yet settled how heavily the assessment should affect promotion decisions. Officials have indicated they intend to learn from the first cycle and refine the process.

That is the public picture. We should not invent more.

Inference: Running realistic assessments with simulations, digital wargames, and AI-enabled components where they are used will also tend to surface information about the technology, the workflows, and how people and machines work together — as well as about the candidates. The tech will not sit there as scenery.

Hope: That the institution deliberately captures and learns from those observations. While we are giving the Pentagon the benefit of the doubt, we are not inventing unreported intent; we are not claiming JWE is an AI research program.

We watch because the first round may teach the institution about the assessment itself — not because anyone has already solved how to assess future generals under AI.

Four things the experiment can teach the rest of us

Since we realize that many organizations, big and small are going through the same transitions, these observations appy to builders and leaders outside the military too.

Recognizing uncertainty is not weakness

Premature confidence on a novel assessment would be more concerning than admitting they do not yet know how to score or weight round one.

The sharper question is: Did they design a process capable of learning whether the assessment works?

There is a useful recursion here. Assess officers. Then assess the assessment. Along the way, they may also learn about the tech and the environment. Uncertainty plus disciplined action can be maturity — not a lack of nerve.

Stop treating every lesson as proof of failure

Not every hard lesson is a failure story. Distinguish three cases:

  • Failure with no learning is waste.
  • Failure with concealment is danger.
  • Failure with learning and adaptation is progress — if the cost stayed inside deliberately accepted boundaries.

Some failures are deadly. Do not romanticize them. Practice where you can, in sandboxes you control.

A locked goal line: Move deliberately enough to learn, safely enough to survive the lesson, and honestly enough to change afterward.

A cheap lesson is a flaw found in an exercise. The principal casualty is overconfidence in the tech.

Lead a system that contains AI — and treat the pair as dynamic

Model change, interface change, retrieval change, policy change, integration change, staff behavior, and trust are not only “measurement contamination.” Leaders operate when those things move. We must remember that yesterday’s demo is not tomorrow’s behavior and handling change is leadership competence.

In a short scenario, the useful moves look ordinary: notice, verify, corroborate, communicate, constrain, continue without, stop, preserve evidence, recalibrate trust, help the organization learn. The test is leading a system that contains AI, not merely “using AI.”

Three things are observable: the human, the technology, and the human-machine team (how people and machines work together under pressure). Assessing humans alone, or tools alone, cannot reproduce what the pair does.

Observability itself has to learn

Observability means keeping enough evidence to understand what happened afterward. Ask: Did we preserve enough of that evidence?

We often discover later that we gathered the wrong signals. Examples include version, prompts or history, retrieved documents, tool calls, config, timestamps, what the human saw and believed, decisions and overrides — and so on.2

Twenty years from now, today’s “enough” may look quaint. That is not automatically failure. Observability is itself a learning discipline. A sandbox teaches what evidence we wish we had had.

The information chain

Traditional supply chains ask: Where did this thing come from?

An emerging information chain asks: Where did this information come from, and what happened to it before it reached me? That is provenance: where the information came from and what happened to it on the way.

One illustrative path: source → database → model → search → summarizer → vendor → analyst → dashboard → decision-maker.

Information can be selected, transformed, and summarized before it reaches a decision-maker. Preserve its provenance along the way. This is an illustrative path, not a universal sequence. Conceptual illustration created with Codex for Renaissance Solutions; CC0-1.0.

The final decision-maker may never knowingly use an AI system and still act on information influenced by several. That is not “inherently bad.” Provenance, monitoring, resilience, and fallback belong to the information chain the way they belong to physical supply chains.3

We are no longer NPCs

An NPC (a scripted character whose rules don’t change) assumes the world’s rules are fixed. In the past, when rules rarely changed, some of us just followed the rules we were given. More and more, that is not a workable option.

Leaders, owners, decision-makers, and people doing the work cannot make that assumption. Some assumptions will go wrong while they are still responsible.

  • A leader still owns the call when the system behaves differently than the brief.
  • An owner still owns the risk when a vendor’s summary was shaped upstream.
  • A decision-maker still owns the action when the dashboard looked clean and was wrong.
  • A person doing the work still owns the stop when something feels off.

The recurring question: The system changed. Something unexpected happened. You are still responsible. What do you do now?

A late note on Kobayashi Maru

For Star Trek fans, in one Starfleet Academy's final commander assessments, there was no winning decision. The test asks what kind of commander you are then.

The modern add is sharper. You do not completely understand every system you command. Some can change. Some of the information helping you may be wrong. What kind of leader are you then?

Sharper still: Can you recognize when the problem you prepared for is no longer the problem you have?

The rules themselves cannot always be assumed fixed — in military settings and in ordinary organizations alike.4

Why we are optimistic

Not because everything is under control. Not because maturity is already proved.

Because of how the Pentagon's experiment is framed publicly so far: learn from the first cycle, refine the assessment, keep the weight of results unsettled until they see how the rollout goes. Those are signs of sober decision-making through the hype — uncertainty acknowledged, experimentation made explicit.

Treating AI as an operational system rather than magic is a broader habit. Adjacent practice points that way — NIST on monitoring deployed systems, and Project Convergence-style experimentation — not as JWE’s public framing.25

A short maturity ladder: happened → recognized → understood why → what should change → changed → tested whether the change worked. A lesson recognized but not fully incorporated is still a step. Transparency matters. Evidence of actual learning matters more.

Recognizing a lesson is a beginning. Organizational learning includes changing practice and testing the result, with evidence that supports another revision when needed. Conceptual illustration created with Codex for Renaissance Solutions; CC0-1.0.

Sober progress is still progress.

Epilogue: role-play the fire before there is smoke

At Renaissance Solutions, we care about practice fires before they’re real — not because every failure is predictable, but because some real ones are expensive, disruptive, or deadly. Use a sandbox where failure is cheap enough to learn from.

The exercise is practice noticing, communicating, deciding who decides, stopping, operating without, preserving evidence, and learning afterward. It is not memorize the correct response. The first experience of uncertainty, conflicting information, unexpected system behavior, and pressure ideally is not the real incident.

The goal is not to avoid every failure. The goal is to make the failures we can safely practice cheap enough to learn from, visible enough to understand, and useful enough that we do not have to learn the same lesson twice.

Footnotes

  1. Natalie Oliverio, “Pentagon adds new ‘warfighter’ evaluation for officers seeking first star,” Military Times, Sep 22, 2026. https://www.militarytimes.com/news/pentagon-congress/2026/09/22/pentagon-adds-new-warfighter-evaluation-for-officers-seeking-first-star/ — Re-verified live 2026-09-26. Service-format detail partly via Fox News internal docs per Military Times (not independently confirmed by the Pentagon as of the article evening). ↩

  2. Adjacent practice only (not a claim about JWE telemetry): Rao et al., “Challenges to the monitoring of deployed AI systems,” NIST.AI.800-4, published Mar 6, 2026. https://www.nist.gov/publications/challenges-monitoring-deployed-ai-systems-center-ai-standards-and-innovation — DOI https://doi.org/10.6028/NIST.AI.800-4 ↩ ↩2

  3. Adjacent practice only (not evidence that JWE follows NIST): NIST AI Risk Management Framework (voluntary; released Jan 26, 2023), https://www.nist.gov/itl/ai-risk-management-framework — and AI RMF Playbook Manage (post-deployment monitoring, third-party risk), https://airc.nist.gov/airmf-resources/playbook/manage/ ↩

  4. Pat Henriques, “The Kobayashi Maru and the Art of Partial Victory,” The Cove (Australian Army), 30 Jun 2026. https://cove.army.gov.au/article/kobayashi-maru-and-art-partial-victory — Cultural cousin for leadership under unwinnable or changing conditions. Not offered as evidence about JWE. ↩

  5. Adjacent institutional cousin only (no causation fold into JWE): Nina Borgeson, “Project Convergence-Capstone 6: FCC’s role in Army experimentation,” army.mil, Jul 24, 2026. https://www.army.mil/article/294098/project_convergence_capstone_6_fccs_role_in_army_experimentation ↩