Overview
Microsoft’s Orca dataset uses GPT-4 to generate detailed reasoning traces for diverse instructions, teaching smaller models to emulate step-by-step thinking. This teacher-student approach significantly narrowed the gap between large and small models.
What’s In It
The dataset contains 5 million instructions paired with detailed explanations from GPT-4 and ChatGPT. Instructions are sampled from the Flan collection with system prompts requesting step-by-step reasoning.
How It’s Used
Orca data trains the Orca model family which demonstrated that 13B parameter models could match GPT-3.5 on many tasks when trained with reasoning traces. The approach influenced subsequent work on knowledge distillation.
Controversies
The approach distills proprietary model capabilities into open models, raising intellectual property questions. Critics argue models trained this way learn to mimic reasoning patterns without truly understanding them.