Rethinking Direct Alignment: Balancing Likelihood and Diversity for Better Model Performance
The problem of over-optimization of likelihood in Direct Alignment Algorithms (DAAs), such as Direct Preference Optimisation (DPO) and Identity Preference Optimisation (IPO), arises when these methods fail to improve model…