arXiv 2305.18290
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
By Rafael Rafailov, Archit Sharma, et al.
Published 2023-05-29
Mindmap
Browse the paper's core ideas, clusters, and relationships in a structured outline.
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with re…