LLMs excel at fitting finetuning data, but are they learning to reason or just parroting🦜? We found a way to probe a model's learning process to reveal *how* each example is learned. This lets us predict model generalization using only training data, amongst other insights: 🧵

The deepseek R1 recipe seems so simple. I’ve been wondering what’s changed from previous RL+reasoning efforts, and found this thread insightful

RT @aviral_kumar2: 🚨 We are organizing an ICLR workshop on self-improving foundation models w/o human supervision at ICLR 2025 in Singapore…

RT @sea_snell: Can we predict emergent capabilities in GPT-N+1🌌 using only GPT-N model checkpoints, which have random performance on the ta…

RT @j_foerst: Learnability for the win! This is one of the lessons that transfers from Curriculum methods in RL directly to LLM training.

@xordrew @setlur_amrith @its_dibya @JacobSteinhardt @svlevine @aviral_kumar2 Potentially because the model has less incentive to change if it has already achieved low loss, though I think there’s some prior work that shows it can happen sometimes if you train for a really long time,e.g. in

@chaochunh Ohh yes you’re right, thanks for catching this!!

@BlancheMinerva Omg thank you!! Will definitely reach out if we end up pursuing this direction 😊

RT @aviral_kumar2: Check out @katie_kang_'s work on understanding memorization vs learning in reasoning! By probing LLMs in training, we…

@chaochunh It's an indicator variable in MaskedAcc, so 1 if perp > p (not memorized) and 0 if perp < p (memorized)

@VarunGodbole @setlur_amrith @its_dibya @JacobSteinhardt @svlevine @aviral_kumar2 With a suboptimal training setup (e.g. hyperparams), models can sometimes directly memorize examples that they may otherwise learn more generalizably if the training setup was better. So there's a tradeoff between direct memorization vs generalizable learning before memorization

@VarunGodbole @setlur_amrith @its_dibya @JacobSteinhardt @svlevine @aviral_kumar2 Once models learn to generate diverse+correct CoTs to a train example, they tend to retain the ability to generate robust predictions, regardless of whether it memorizes the example later in training

@SadhikaMalladi Thanks for sharing! I haven't seen this before

@ahatamiz1 Thanks! We haven't, but it would definitely be interesting to better understand the relationship btw architecture and generalization

@chaochunh 1) yes! in calculating pre-mem acc, we mask out accuracy when perplexity is too low 2) it makes the scale of numbers slightly easier to work with

@BlancheMinerva Thanks! It would definitely be interesting to study the learning dynamics of pretraining as well

@oier_mees Thanks Oier!

RT @avisingh599: Exciting couple of days for reasoning research: Procedural Knowledge in Pretraining Drives Reasoning in Large Language Mo…

RT @svlevine: An intriguing new result from @katie_kang_: after training long enough, LLMs will reproduce training examples exactly (not su…

@brianckwu Thank you :)