RS Glossary
Direct Preference Optimization
DPO stands for direct preference optimization, a training method that teaches a model from preference comparisons without running full reinforcement learning.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 2
- Direct Preference Optimization
- DPO
Remember it this wayDPO learns directly from preferred-versus-rejected response pairs.
In plain language
Direct Preference Optimization, or DPO, fine-tunes a language model from comparisons that indicate which of two responses is preferred. It optimizes the model against those preferences without first training a separate reward model and then running reinforcement learning.
Why it matters
DPO can simplify preference-based post-training, but the result still depends on who supplied the comparisons, what choices they saw, and what the preference data rewards or omits.
Example
Reviewers compare two answers to the same prompt. DPO uses the preferred and rejected responses to adjust the model toward patterns found in the preferred set.
Common misconception
DPO does not remove human judgment from alignment. It changes how preference data is used; it does not make the preferences objective or universally correct.
Related terms
Sources
- Rafael Rafailov and colleagues, “Direct Preference Optimization: Your Language Model Is Secretly a Reward Model”introduces DPO as direct optimization from preference comparisons without the standard RLHF reward-model loop.
Learn more
- Compare [[rlhf|RLHF]], [[fine-tuning|Fine-Tuning]], [[alignment|Alignment]], and [[human-review|Human Review]] to place Direct Preference Optimization in the surrounding system.
- Follow the cited source material above for the method, standard, or research context behind this draft.
Public history
23 recorded events · 23 public summaries
A potential shared concept was discussed and selected for initial glossary drafting.
The captured candidate received a complete initial draft and entered section-level human review.
Radar reviewed the initial definition draft for human consideration.
Radar reviewed the initial plain-language-explanation draft for human consideration.
Radar reviewed the initial why-it-matters draft for human consideration.
Radar reviewed the initial example draft for human consideration.
Radar reviewed the initial common-misconception draft for human consideration.
Radar reviewed the initial remember-it-this-way draft for human consideration.
Radar reviewed the initial related-terms draft for human consideration.
Radar reviewed the initial sources draft for human consideration.
Radar reviewed the initial learn-more draft for human consideration.
Definition review recorded as approve.
Plain-Language Explanation review recorded as approve.
Why It Matters review recorded as approve.
Example review recorded as approve.
Common Misconception review recorded as approve.
Remember It This Way review recorded as approve.
Related Terms review recorded as approve.
Sources review recorded as approve.
Learn More review recorded as approve.
Sources review recorded as replace.
Radar verified the learner-facing sources after personal origin provenance was removed.
All nine current wording approvals were reconciled to Approved term status.
Source: vault/Glossary/terms/direct-preference-optimization/term.md · sha256:47d2229e2512777e0507e441b1f6311129da3edda3d4ebf8001d369caf7c6420 · Release: rs-glossary-glossary-team-poc-0.5.5-2026-09-02