arXiv 2305.18290
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
By Rafael Rafailov, Archit Sharma, et al.
Published 2023-05-29
Citation lineage
Review the prior work and downstream research connected to this paper.
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with re…