Invited Talk by Dr. Sayak Ray Chowdhury

Abstract

Direct Preference Optimization (DPO) has emerged as a popular alternative to reinforcement learning from human feedback (RLHF) for aligning models trained with purely supervised objectives using preference data. In this talk, I show that DPO implicitly solves a statistical estimation problem over reward functions induced by a parametric policy class, which can become misspecified when the true reward lies outside the class, leading to failure modes such as preference order reversal and degraded policy performance. We then analyze the local geometry of two-stage RLHF updates and relate them to natural gradient steps in policy space. This perspective motivates AuxDPO, a principled modification that introduces auxiliary variables to better approximate RLHF and mitigate misspecification. Experiments on language model alignment tasks demonstrate improved robustness and performance.