Neural architecture (replacing attention), learning algorithms (replacing backprop), and hardware architecture (replacing von Neumann). Not really new axes, but newly effective. Stay tuned.

Ok we've read a lot about test-time compute being the new scaling axis, but what's the next scaling axis?

🚀🔬 Introducing Agent Laboratory: an assistant for automating machine learning research Agent Laboratory takes your research ideas and outputs a research paper and code repository, allowing you to allocate more effort toward ideation rather than low-level coding and writing 🧵

Why do people think this is real? A "transformer" ASIC makes no sense. BS=1 is ram bandwidth bound. BS=big is FLOPS bound. And NVIDIA "Transformer Engine" has everything you might need re: scaling small dtypes. Hype is disgusting. Stop falling for it.

The French Polymarket whale commissioned polls with a specific alternate methodology, the “neighbor method” 1. What a baller 2. What a killer example of how betting markets can surface contrarian, high quality signals

Some examples from our work: Hebbian cortical-like microcircuits enable the most efficient and (still?) best performing, non-backprop deep learning algorithm Spikes speed up learning and inference in sequence processing Efference copies improve and explain self-supervised learning Short-lived associative plasticity without training outperforms trained RNNs, LSTMs etc in certain scenarios Meta-learned short-term plasticity outperforms them in more scenarios Several examples of nano-devices performing neural computations extremely efficiently. etc. Would they have been figured out by engineering alone? Maybe, at some point. But computational neuroscience was certainly our origin.

I'd like to teach a paper which shows how a fact about the brain materially improved an AI system in a way that is unlikely to have been figured out by engineering alone. I haven't been able to find a single example of this. Suggestions welcome.