Review Direct Preference Optimization

DPO stands for direct preference optimization, a training method that teaches a model from preference comparisons without running full reinforcement learning.

Direct Preference Optimization, or DPO, fine-tunes a language model from comparisons that indicate which of two responses is preferred. It optimizes the model against those preferences without first training a separate reward model and then running reinforcement learning.

DPO can simplify preference-based post-training, but the result still depends on who supplied the comparisons, what choices they saw, and what the preference data rewards or omits.

Reviewers compare two answers to the same prompt. DPO uses the preferred and rejected responses to adjust the model toward patterns found in the preferred set.

DPO does not remove human judgment from alignment. It changes how preference data is used; it does not make the preferences objective or universally correct.

DPO learns directly from preferred-versus-rejected response pairs.

* [[rlhf|RLHF]] * [[fine-tuning|Fine-Tuning]] * [[alignment|Alignment]] * [[human-review|Human Review]]

* [Rafael Rafailov and colleagues, “Direct Preference Optimization: Your Language Model Is Secretly a Reward Model”](https://arxiv.org/abs/2305.18290) — introduces DPO as direct optimization from preference comparisons without the standard RLHF reward-model loop.

* Compare [[rlhf|RLHF]], [[fine-tuning|Fine-Tuning]], [[alignment|Alignment]], and [[human-review|Human Review]] to place Direct Preference Optimization in the surrounding system. * Follow the cited source material above for the method, standard, or research context behind this draft.