It seems to me that there just isn’t a lot of incentive for compression in LLMs (sure you have some low norm regularizers but it’s not clear to me how this translates to representation compression/MDL of the content after decoding), which seems required/equivalent to finding novel abstractions. Model scale is actually counterproductive here (assuming classical transformer arch). This is also clear from the many papers showing failures/inefficiencies in the algos transformers learn from raw data (eg edge of chaos paper). Many directions here that are under-explored imo.

RT @HHMIJanelia: 📢 Submissions are now open for the CellMap Segmentation Challenge. Build the best method for segmenting cellular organel…

RT @JohanWinn: 🚀 We are looking for Image Data Scientists to join our mission to advance connectomics towards whole brain scale for humans…

@cppape Good stuff, congrats!

If anyone had bothered to read the extensive prior work on unsupervised disentangled representation learning, this crime could have been avoided.

@doomslide Vision still doesn’t actually work & we are lacking strong base models that can do efficient RL for everything but math and coding.

@doomslide RL scaling is pure compute though. Not even data constrained anymore, feels like this isn’t factored in properly.

RT @Miles_Brundage: Stargate + related efforts could help the US stay ahead of China, but China will still have their own superintelligence…

@VictorTaelin In case it helps, my unfiltered thoughts were: Amazing stuff, but can’t integrate it into my current research without significant work. So the risk reward ratio is off. Have to choose between replicating r1 or playing around with this. Gotta de-risk it imo, too much going on.

RT @Dorialexander: My main take away of the Deepseek paper is not scientific but organizational: we need an European industrial plan in AI…

@somewheresy Sounds similar to the general problem of not diverting too much from the base distribution that is addressed in post training (eg rlhf) by staying close to the base distribution. So you could try to regularize with a reference from a base output/model.

@JustinLin610 A QwQ paper? 🙏

@GarrettPetersen Would recommend at least 1B parameters for realistic wailing.

@cloneofsimo Is this really a good way to look at this? Solving a trivial QA in token embedding space also gets you ~zero human accuracy.

@RajeshBhayana_ On it

„as we know it“ carries all the weight here.

@far__el Some don’t, e.g. those who recycle this take in perpetuity.

@CRSegerie WDYM? Have you used llama 3? There is no shot that model can do any of the things you are afraid of. This type of fear mongering with zero technical backup is pretty destructive, not good.

@signulll The inverse may actually be true. Long term strategic thinking and managing large groups of agents that have short term planning capabilities is plausibly the highest impact position in the new world.