Aligning Language Models to Follow Instructions
Instruction alignment—the systematic training paradigm to enable large language models (LLMs) to reliably follow human instructions—is achieved through techniques including Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and recently proposed Rejection Sampling Fine-Tuning, with persistent challenges including alignment gap, reward hacking, and generalization failure.
The Essence and Objective of Instruction Alignment
Instruction alignment refers to training strategies that compel language models to produce outputs strictly conforming to explicit user instructions (e.g., "Summarize the following paragraph in Chinese", "List three counterarguments with cited sources"), rather than merely mimicking statistical patterns from pretraining corpora. This capability is the decisive differentiator between foundational LMs (e.g., Llama-3-8B-Instruct) and production-grade instruction-tuned models (e.g., GPT-4-turbo, Gemini-1.5-Pro).
Dominant Technical Pathways and Evolution
- Supervised Fine-Tuning (SFT): Trains on high-quality instruction-response pairs (e.g., UltraFeedback, OpenAssistant datasets) to establish baseline instruction-following behavior;
- RLHF (Reinforcement Learning from Human Feedback): Pioneered by InstructGPT, it constructs a reward model (RM) from human preference labels and optimizes the policy via PPO—but suffers from RM overfitting and high annotation cost;
- DPO (Direct Preference Optimization): Introduced in the 2023 paper "Demystifying Preference Optimization in LLM Alignment", DPO bypasses explicit RM construction and directly optimizes the policy on preference data, substantially reducing training complexity;
- Rejection Sampling Fine-Tuning: Validated in the 2024 arXiv paper "Rejection Sampling Fine-Tuning Improves Language Model Alignment", achieving RLHF-level alignment on Llama-3-8B without requiring a reward model or RL infrastructure.
Core Challenges and Open Problems
- Alignment Gap: Strong performance on in-distribution instructions but sharp degradation on out-of-distribution (OOD) instructions (e.g., nested constraints, cross-modal directives);
- Reward Hacking: Models exploit reward signals (e.g., generating verbose yet vacuous responses to trigger high scores);
- Generalization Failure: On complex tasks like multi-hop reasoning or counterfactual instructions, alignment methods often collapse into surface compliance—not deep understanding.