Review RLHF

RLHF stands for reinforcement learning from human feedback, a method that uses human preferences to help train a model toward more desirable responses.

Reinforcement Learning from Human Feedback, or RLHF, uses human judgments to train a reward signal and then adjusts a model to produce outputs that score better under that signal.

RLHF can make outputs more useful or aligned with stated preferences, but the result reflects the reviewers, prompts, reward design, and optimization choices used in the process.

Reviewers rank several answers to the same prompt. A reward model learns from the rankings, and reinforcement learning adjusts the language model toward higher-scoring responses.

RLHF does not give a model human values or eliminate harmful behavior. It optimizes against a learned approximation of selected human feedback.

Human preferences train a reward; the reward steers the model.

* [[alignment|Alignment]] * [[human-review|Human Review]] * [[fine-tuning|Fine-Tuning]] * [[direct-preference-optimization|Direct Preference Optimization]]

* [Long Ouyang and colleagues, “Training Language Models to Follow Instructions with Human Feedback”](https://arxiv.org/abs/2203.02155) — documents supervised fine-tuning, reward-model training, and reinforcement-learning stages used in InstructGPT.

* Compare [[alignment|Alignment]], [[human-review|Human Review]], [[fine-tuning|Fine-Tuning]], and [[direct-preference-optimization|Direct Preference Optimization]] to place RLHF in the surrounding system. * Follow the cited source material above for the method, standard, or research context behind this draft.