Overview
OpenOrca is a community project that openly reproduces Microsoft’s Orca approach, using GPT-4 to generate explanatory responses at scale. It provides data and methodology for training small models that punch above their weight through detailed reasoning.
What’s In It
OpenOrca contains over 1 million entries from the Flan collection augmented with GPT-4 generated responses including step-by-step explanations. Each entry includes the instruction, a system prompt requesting reasoning, and GPT-4’s response.
How It’s Used
OpenOrca data is used in fine-tuning mixtures for high-performing open models. It enables researchers without GPT-4 API budgets to leverage the Orca methodology. It is commonly combined with other instruction datasets.
Controversies
Generating training data via GPT-4 API is expensive and creates OpenAI dependency. The approach may violate terms of service. Questions remain about whether explanation-based distillation truly transfers reasoning ability.