🌟Announcing NeurIPS spotlight paper on the transition from lazy to rich🔦 We reveal through exact gradient flow dynamics how unbalanced initializations promote rapid feature learning co-led @AllanRaventos and @ClementineDomi6 @FCHEN_AI @klindt_david @SaxeLab @SuryaGanguli

RT @MasonKamb: Excited to finally share this work w/ @SuryaGanguli. Tl;dr: we find the first closed-form analytical theory that replicates…

Come check out our #NeurIPS2024 spotlight poster on feature learning tomorrow! 📍East Exhibit Hall A-C #2102 📅Thu 12 Dec 4:30 p.m. — 7:30 p.m. PST

@ziv_ravid @GalantiTomer @NiketPatel91154 Another related work that studies the relationship between SGD and the rank & sparsity of weights (although seems your analysis of local rank decreasing doesn't depend on noisy gradients?)

RT @klindt_david: Great job, it was an honor being part of this amazing project! Congrats to the team 💪

Also, big shoutout to @yasamanbb, @CPehlevan, and @HSompolinsky for coordinating last year's 'Deep Learning from Physics and Neuroscience' program @KITP_UCSB. Our amazing team met there, and this project is a direct result of the conversations we had!

Applying our function space analysis to shallow ReLU networks, we find that rapid feature learning occurs from unbalanced initializations that promote faster learning in early layers, driving a large change in activation patterns, but a small change in parameter space

We find three regimes in function space: 1. lazy akin to linear regression, 2. rich akin to silent alignment (Atanasov et al. 2021), 3. delayed-rich initially lazy followed by rich We extend this analysis (with mirror flows and implicit biases) to wide & deep linear networks

We derive exact gradient flow solutions for a minimal two-layer linear model displaying lazy and rich learning, which reveals that the relative scale between layers influence feature learning through conserved quantities that constrain the geometry of learning trajectories

Reproducing Fig.1 in Chizat et al. 2019 we find that even at small overall scale, the relative scale between layers can transition the network between rich and lazy learning and the best generalization occurs at small scale and large relative scale!

Really cool theory project on feature learning. If you are at the HiLD workshop @icmlconf check it out!

RT @AllanRaventos: Interested in exactly solvable models of learning dynamics and implicit bias? Come check out our "Get Rich Quick" poster…

RT @DebOishi: Reminder! Happening Tomorrow! @ELLISforEurope

To get the zoom link and get notified about other interesting talks check out

RT @DebOishi: We are delighted to announce our next speakers for the @ELLISforEurope RG are fr @Stanford, the authors of the NeurIPS2023 pa…

@ItsNeuronal Neat! In part of our work we track empirically the degree of redundancy in networks through training. Maybe your metric would help. Let's chat about it tomorrow at the poster session!