Review Calibration

Calibration describes how well stated confidence matches observed correctness across comparable cases. A well-calibrated system that assigns 80 percent confidence should be correct about 80 percent of the time on cases receiving that confidence.

Calibration asks whether confidence has earned its meaning. It compares what a person or system says about uncertainty with what happens over many relevant examples.

People often treat confident language as evidence of reliability. Calibration provides a record-based check. It can reveal overconfidence, underconfidence, and conditions where a confidence signal should not guide a decision.

A classifier labels 100 comparable cases with 90 percent confidence, but only 65 labels are correct. The classifier may still be useful, but its confidence is poorly calibrated for those cases.

Calibration is not the same as accuracy, and it is not the same as the simpler question of whether one answer is correct. Confidence vs. Correctness separates two properties of a single answer. Calibration measures whether confidence levels match correctness rates across a defined set of comparable cases. A system can therefore be accurate overall and still be poorly calibrated.

Confidence should match the record.

* [[confidence-vs-correctness|Confidence vs. Correctness]] * [[verification|Verification]] * [[epistemic-humility|Epistemic Humility]] * [[observation|Observation]]

* [Chuan Guo and colleagues, “On Calibration of Modern Neural Networks”](https://proceedings.mlr.press/v70/guo17a.html) — calibration as agreement between predicted confidence and empirical correctness, and its distinction from accuracy.

* [RS AI Learning Lab, “Designing Systems That Can Admit They Are Wrong”](../../../../modules/designing-systems-that-can-admit-they-are-wrong/module.md) * [RS AI Learning Lab, “When the Bot Sounds Right but Is Not”](../../../../modules/when-the-bot-sounds-right-but-is-not/module.md)