Excited to share new work from @GoogleDeepMind / @GoogleResearch on “Robust Preference Optimization through Reward Model Distillation”.

RT @stats_stephen: Important topic, but this is more of a quick-start guide. For cutting-edge research on LLM evals, see these papers usin…

@ml_angelopoulos Awesome stuff!

RT @ml_angelopoulos: 🚨 New Textbook on Conformal Prediction 🚨 “The goal of this book is to teach the reader about…

@raymin0223 @GoogleDeepMind Thank you for all the great work Sangmin!

RT @aviral_kumar2: This work was led by the amazing @setlur_amrith during his internship at Google Research. With @nagpalchirag, @adamjfisc…

RT @aviral_kumar2: 🚨New paper led by @setlur_amrith on process rewards for reasoning! Our PRMs that model specific notion of "progress" re…

RT @setlur_amrith: 🚨 Exciting new results with dense process reward models (PRMs) for reasoning. Our PRMs scale ✅ search compute by 1.5-5x…

@GoogleDeepMind @GoogleResearch @ml_angelopoulos Checkout the paper for more details! Fun work done together with a great team: @maynez_joshua, @rhofour, @bhuwandhingra, @amirgloberson, and @professorwcohen .

RT @raymin0223: 🚨Check out our new paper, Block Transformer! We propose an efficient architecture with Global-to-Local language modeling.…

@amritsinghbedi3 @SOURADIPCHAKR18 @GoogleDeepMind @GoogleResearch Yes, the issue is shifted to the RM now. But the point we make is that these can be easier to explicitly regularize/clip/ train multiple/etc. w/o having to derive a new obj for each variant. Also, as it’s real-valued, the RM won’t give p(y_1 > y_2 | x) = 0 for any (y_1, y_2, x).

@amritsinghbedi3 @SOURADIPCHAKR18 @GoogleDeepMind @GoogleResearch This is motivated by situations where the Bradley-Terry MLE is degenerate, which happens in most real datasets where we see 1, or just a few, sampled prefs for any given pair. We focus on binary prefs; depends, but prob. true for most practical annotation schemes with randomness.

@SOURADIPCHAKR18 @GoogleDeepMind @GoogleResearch They can be created in any way, don’t need to follow any rules or forms. In the paper, we used an ensemble of RMs with different inductive biases (not all of which may be correct choices). See also for some recent RM ensemble construction techniques.

@TengX6 @GoogleDeepMind @GoogleResearch pi*_{r_tgt} just denotes the policy that maximizes the exp. reward according to r_tgt, subj. to the KL penalty. That is, pi*_{r_tgt} = argmax_pi E_pi(y|x) [ r_tgt(x,y) ] - beta * KL(pi || pi_ref). (29) plugs r_tgt written in terms of pi*_{r_tgt} back into the main objective.

RT @TechnionLive: 🎉Exciting news! Three young faculty members from @TechnionLive have been awarded the 2024 Krill Prize for Excellence in S…