Subhash Kantamneni
@thesubhashk
Followers
755
Following
20
Statuses
45
@mit Tegmark group. mech interp & ai4science
Boston, MA
Joined February 2024
(1/N) LLMs represent numbers on a helix? And use trigonometry to do addition? Answers below 🧵
22
158
927
More great work from the @tegmark lab introducing harmonic loss, which maintains the performance of cross entropy while producing more interpretable (and beautiful) representations!
1/9 🚨 New Paper Alert: Cross-Entropy Loss is NOT What You Need! 🚨 We introduce harmonic loss as alternative to the standard CE loss for training neural networks and LLMs! Harmonic loss achieves 🛠️significantly better interpretability, ⚡faster convergence, and ⏳less grokking!
0
4
9
Really cool way to think about improving SAEs! Matt’s really bright and a great member of the @tegmark group
(1/11) New paper! “Low-rank adapting models for Sparse Autoencoders.” While SAEs find interpretable latents, they hurt downstream behavior—e.g. using TopK SAE activations on GPT-4 mimics a model trained w/ 10% compute. Our fix? Adapt the model for the SAE, not just vice versa.👇
0
0
5
awesome to see my friends @NithinParsan and @johnyang100 doing mech interp research on bio models! excited to see where it goes
YC F24's @ReticularAI makes protein AI models controllable and interpretable to help steer protein design with limited biological data, reducing costly validation cycles. Congrats on the launch, @NithinParsan and @johnyang100!
1
0
7
(6/N) Read our blog post here: Work was done as part of the two week sprint in @NeelNanda5's MATS stream with my excellent co-first author, @JoshAEngels
0
0
9