RS Glossary
Evaluation Gap
An evaluation gap is the difference between what an evaluation would need to test to support a real-world claim and what the evaluation actually measures.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 1
- Evaluation Gap
- AI Evaluation Gap
Remember it this wayThe score answers less than the claim.
In plain language
A score usually covers a slice of behavior under chosen data, metrics, users, and conditions. The evaluation gap is what remains untested when someone stretches that score into a broader claim about usefulness, safety, fairness, or readiness for a particular setting.
Why it matters
A narrow evaluation can create false confidence even when its numbers are accurate. Missing languages, users, failure modes, tool interactions, time pressure, or real consequences can matter more than the measured benchmark once the system is deployed.
Example
A support assistant scores well on factual accuracy for clean English questions. It is then called production-ready even though no evaluation covered multilingual users, prompt injection, tool failures, response latency, or harmful refusals in the actual support workflow.
Common misconception
An evaluation gap is not simply a low score or a lack of many tests. A large benchmark suite can still leave a serious gap if it measures the wrong conditions or values for the claim being made. The gap is between coverage and claim, not between a score and perfection.
Related terms
Sources
- Hutchinson et al., “Critical Evaluation Gaps in Machine Learning Practice”documents the gap between the broad concerns needed for responsible system use and the narrow predictive behaviors commonly evaluated.
- NIST AI RMF Playbook, “Manage”calls for evaluation under deployment-like conditions and continuing post-deployment monitoring as context and risk change.
Learn more
- Write the deployment claim first, then use [[verification|Verification]] to ask what evidence would actually support it.
- Record tested and untested conditions through [[capability-provenance|Capability Provenance]] and a [[model-card|Model Card]].
Public history
30 recorded events · 30 public summaries
A potential shared concept was discussed and selected for initial glossary drafting.
The captured candidate received a complete initial draft and entered section-level human review.
Radar reviewed the initial definition draft for human consideration.
Radar reviewed the initial plain-language-explanation draft for human consideration.
Radar reviewed the initial why-it-matters draft for human consideration.
Radar reviewed the initial example draft for human consideration.
Radar reviewed the initial common-misconception draft for human consideration.
Radar reviewed the initial remember-it-this-way draft for human consideration.
Radar reviewed the initial related-terms draft for human consideration.
Radar reviewed the initial sources draft for human consideration.
Radar reviewed the initial learn-more draft for human consideration.
Definition review recorded as approve.
Plain-Language Explanation review recorded as approve.
Why It Matters review recorded as approve.
Example review recorded as approve.
Common Misconception review recorded as approve.
Remember It This Way review recorded as approve.
Related Terms review recorded as approve.
Sources review recorded as approve.
Learn More review recorded as approve.
All wording and media review gates are approved; the term is now canonical.
Definition review recorded as approve.
Plain-Language Explanation review recorded as approve.
Why It Matters review recorded as approve.
Example review recorded as approve.
Common Misconception review recorded as approve.
Remember It This Way review recorded as approve.
Related Terms review recorded as approve.
Sources review recorded as approve.
Learn More review recorded as approve.
Source: vault/Glossary/terms/evaluation-gap/term.md · sha256:fbef8c0c0512000156f08cbdbe83e60a9de0d673d7ab5b0f46b595ab98f7b4dc · Release: rs-glossary-glossary-team-poc-0.5.5-2026-09-02