Overview
DPO-Mix refers to curated mixtures of human preference datasets assembled for Direct Preference Optimization training. These collections combine multiple sources to align language models with human values without requiring online reinforcement learning.
What’s In It
Typical DPO-Mix datasets contain approximately 60,000 preference pairs combining sources like Anthropic HH-RLHF, Stanford SHP, OpenAI summarization preferences, and UltraFeedback. Each example has a prompt with a preferred and rejected response.
How It’s Used
DPO-Mix datasets enable alignment training through Direct Preference Optimization which optimizes models to prefer human-rated better responses without a separate reward model. Models like Zephyr and Tulu 2 use such mixtures.
Controversies
Combining preference datasets from different sources with different annotation guidelines can create inconsistent training signals. Human preferences are subjective and culturally conditioned. Binary preference pairs lose nuance about why responses are preferred.