RLHF trains a reward model on human preference rankings, then uses reinforcement learning to fine-tune language models toward that reward signal. OpenAI's InstructGPT research demonstrates the three-step method in practice: human labelers preferred outputs from the 1.3-billion-parameter InstructGPT over the 175-billion-parameter GPT-3, a model with 100 times more parameters.
What Is RLHF, and Why Align Language Models to Human Values?
RLHF trains a reward model with direct human feedback, then uses it to optimize an AI agent's performance through reinforcement learningCITE:E7. IBM defines the technique this way, describing it as a machine learning method in which a reward model is trained on human feedback and subsequently used to improve an AI agent's behavior through reinforcement learningCITE:E7. Hugging Face states that RLHF has enabled language models to begin aligning a model trained on a general corpus of text data to complex human valuesCITE:E6.
What Is the First Step, and What Role Does Supervised Learning Play?
The first step fine-tunes GPT-3 on labeler-written demonstrations of desired behaviorCITE:E1. In the InstructGPT paper published March 4, 2022, OpenAI describes collecting a dataset of labeler demonstrations of the desired model behavior and using it to fine-tune GPT-3 with supervised learningCITE:E1. This supervised model becomes the starting point for the next two steps.
What Is the Second Step — How Is a Reward Model Trained From Human Ranking Data?
The second step trains a reward model on human rankings of model outputsCITE:E2. OpenAI collects a dataset of rankings of model outputs and uses it to further fine-tune the supervised model with reinforcement learning from human feedbackCITE:E2. Hugging Face explains that the underlying goal of this reward model is to take in a sequence of text and return a scalar reward that numerically represents the human preferenceCITE:E4.
What Is the Third Step — How Does PPO Apply in Reinforcement Learning Fine-Tuning?
The third step applies Proximal Policy Optimization, or PPO, to fine-tune the language model against the reward signalCITE:E5. Hugging Face describes this step as fine-tuning some or all of the parameters of a copy of the initial language model using a policy-gradient reinforcement learning algorithm, PPOCITE:E5. This step builds directly on the ranking data collected in step two, since OpenAI uses that dataset to further fine-tune the supervised model through RLHFCITE:E2.
What Did InstructGPT Show Compared to GPT-3?
Human labelers preferred outputs from the 1.3-billion-parameter InstructGPT model over the 175-billion-parameter GPT-3CITE:E3.
| Model | Parameters | Labeler Preference |
|---|
| InstructGPT | 1.3B | Preferred |
| GPT-3 | 175B | Less preferred |
| Parameter gap | InstructGPT has 100x fewer parameters than GPT-3 | — |
OpenAI reports that despite having 100 times fewer parameters, the 1.3B InstructGPT model's outputs were preferred to those of the 175B GPT-3CITE:E3.
What This Means
The three-step process OpenAI describes in the InstructGPT paper — supervised fine-tuning on demonstrationsCITE:E1, reward model training on ranked outputsCITE:E2CITE:E4, and PPO-based reinforcement learningCITE:E5CITE:E2 — matches the general RLHF framework defined by IBMCITE:E7 and the alignment goal described by Hugging FaceCITE:E6. The InstructGPT result, where a 1.3B-parameter model outperformed a 175B-parameter model on human preference despite 100 times fewer parametersCITE:E3, is the concrete outcome tied to that three-step pipeline in this evidence set.
Author's Take・林紀旭 James Lin
The InstructGPT result is the strongest evidence here for why RLHF matters as a training architecture, not just a fine-tuning add-on: a 1.3-billion-parameter model beat a 175-billion-parameter model on human preference despite carrying 100 times fewer parameters. That gap did not come from scale — it came from the three-step pipeline itself: supervised fine-tuning on demonstrations, a reward model trained on ranked outputs, and PPO optimization against that reward signal. For anyone weighing investment in an RLHF pipeline against simply scaling a base model, the metric worth tracking is preference-win rate per parameter, not raw parameter count or benchmark scores. Whether the same 100x pattern holds outside the specific InstructGPT setup described here is a question this evidence set does not answer.