Model Distillation in the API: Fine-tuning Cost-Efficient Models Using GPT-4 Outputs on the OpenAI Platform

Core Method: API-Driven Supervised Distillation

High-quality instruction-response pairs are generated at scale via OpenAI API calls (e.g., gpt-4-turbo or gpt-4o), then used as supervision signals to fully fine-tune or LoRA-fine-tune target small models—such as gpt-3.5-turbo, llama-3-8b-instruct, or locally deployed Phi-3-mini.

Key Technical Characteristics

• Zero-shot teacher signals: Human annotation is bypassed; distillation datasets are built solely from strong generalization outputs of GPT-4-class models;
• Native platform support: OpenAI’s fine_tuning API (v2) and training_file upload interface natively accept JSONL-formatted distillation datasets;
• Quantifiable metrics: On AlpacaEval 2.0, llama-3-8b-instruct distilled with gpt-4-0613 outputs achieves 72.3% win rate (vs. gpt-4-0613), a +14.6-point gain over the base model;
• Cost structure optimization: Compared to continuous gpt-4 API usage ($5/1M input tokens + $15/1M output tokens), inference cost drops to <$0.1/1M tokens on self-hosted A100 80GB GPUs.