WebGPT: Improving the factual accuracy of language models through web browsing
OpenAI fine-tuned GPT-3 (davinci) using supervised learning on browser interaction demonstrations and RLHF on human preference data, significantly improving factual accuracy on open-ended questions—without architectural changes.
Core Method
WebGPT builds upon GPT-3 (specifically the davinci model), introducing no new parameters or architecture. Instead, it learns to simulate human-like text-based web browsing behavior—including search queries, link selection, page scrolling, and snippet extraction—via two-stage training:
- Supervised Fine-Tuning (SFT): Trained on human-written browser interaction trajectories (search → browse → cite) collected via a custom text-based browser interface;
- Reinforcement Learning from Human Feedback (RLHF): A reward model is trained on human preference comparisons between answer-evidence pairs, then used to optimize the policy via PPO.
Key Results
- Achieves higher factual accuracy than baseline GPT-3 (
davinci) and InstructGPT (text-davinci-001) on RealNews dataset and ELI5; - Generates answers with verifiable web citations (URL + excerpt), enabling traceability;
- Runs entirely offline—the simulated browsing occurs over a static, filtered snapshot of Common Crawl and Wikipedia, ensuring reproducibility and safety.
Technical Positioning
This work represents an early, influential exploration of external knowledge integration for LLMs, predating the widespread adoption of RAG (Retrieval-Augmented Generation). It laid methodological groundwork for later systems such as OpenAI’s WebPilot and Google’s Search-Google-Gemini integration. Published on arXiv:2112.09861.