@katie_kang_ @setlur_amrith @its_dibya @JacobSteinhardt @svlevine @aviral_kumar2 It's really interesting that memorization doesn't displace existing generalization. But I guess the converse, where a model learns a generalizable solution after memorizing, doesn't really happen? Any thoughts on why?

@nickcdryan @jxmnop Do you know if this was ever updated for 2024 era models? I wish I could see something like this for mamba, hyena, xlstm, even RAG if you stretch the premise a little.

@RichmanRonald @francoisfleuret I wonder how bad that actually is. There's the standard intuition built around attention being a soft information router, but you can also just call it a reasonable way to factorize a giant matrix.

@francoisfleuret A nice bit of ML lore is that in the original AlexNet, splitting the network to fit across GPUs caused one half to learn texture patterns and the other to learn color patterns. Apparently the effect was pretty consistent.

@owl_poster I kind of want to try more comparisons here, maybe even just with AlphaFold2's internal embeddings. Isn't pLDDT being high for both disordered and fake proteins completely correct behavior?

@NikoMcCarty In the essay on low Reynolds numbers, there's a really interesting note on how almost all liquids have viscosity higher than water's. Did anyone get around to explaining that in the last 50 years?

@andrewwhite01 This makes a lot more sense if you know that lean mass hyper responders are a group that gains cholesterol on a ketogenic diet. Oreos are pretty darn un-ketogenic.

@evanjconrad I've been working on tuning protein language models for iterative design, it's an academic project, but a pretty interesting one.

@francoisfleuret This was the key equation for ballpark estimates. Bottom line is that if you try to improve things like clock rate, power use, etc, error rates increase exponentially. On the upside, acceptable error rates were listed as like 1/computer/year, so lots of OOM to work with.

@francoisfleuret That said, that's if you're keeping normal architectures and just tolerating some low importance bit flips. I'd imagine you can get a lot more creative.

@TimothyDuignan I love this area, and the whole general idea of noisy computing. Random example I liked was oscillator computing: where you just let coupled oscillators converge on a minimum energy ensemble.

@owl_poster Have you seen any good overviews of the types of PPI databases? It's always been embarrassingly unclear to me what interactions specifically mean, and how much variance/false positives you'd get from different experimental methods.

@kotsoft Jello is good enough for me- thanks!

@kotsoft I was just reading that paper! Quick question, do you think it's suitable for modeling substances closer to solid than fluid? I wasn't sure how far you could push the model away from fluid behavior before it would degenerate.

@samsinai You could cover the interpolation space reasonably well by just sampling from a HMM profile for the family. I'm curious how good that'd be as a baseline for e.g. GFP mutation. I'd guess worse, but maybe not by as much as we'd like.

@DmitryRybin1 @du_yilun I've never heard that, could I literally solve sudoku problems with a convex solver?

@TimothyDuignan Awesome article, thanks for sharing! It's funny how much more relevant this feels post AF-3. At this point I'm definitely wondering what point we'll settle on on the continuum between actual MD and direct ML prediction for the harder problems in molecule design.

@TimothyDuignan It always surprises me that diffusion models and alphafold are in some sense 'SOTA' models for NNPs, since they're purely generative models with no real physics baked into training. Have you seen anyone integrate them with MD in a convincing way?