I looked at AIME problems and one thing strikes me. All problems are about computing a number. This is a tiny part of math. I was trained a a mathematician in France. And I almost never had to solve a problem of that kind. All the math work was about proving mathematical properties of mathematical objects. For instance, prove that a given group is isomorphic to another given group. This is to say that getting good at computing numbers specified by some mathematical setting is not the same as getting good at math in general. It is definitely part of math, but only a tiny part of math. There is no wonder AI focuses on number finding math problems. It is because checking the result is simple. Tackling the full spectrum of math requires a much more complex result checking machinery (formal proof checker) It is also interesting to note that AI math benchmarks only care about the final number. If that number was accidentally found via a flawed mathematical proof, then it is still considered a success.

the government runs on elon standard time now

I still haven't heard a good answer to this question, on or off the podcast. AI researchers often tell me, "Don't worry bout it, scale solves this." But what is the rebuttal to someone who argues that this indicates a fundamental limitation?

MCP has been on a roll the past two weeks. Support added to both Cursor and Windsurf. Block built their agent's extensions on top of MCP. And many other major partners are working on adding MCP to their apps right now. MCP is truly the community's integration protocol.

Worst financial decision in history

Existing models failing at your task? Just drop a benchmark and watch the LLM providers trip over themselves to beat it. Problem solved in months.

LLMs are starting to have personalities. User: How are you? GPT-4o: Responds with 4 rocket emojis 🚀 Deepseek-R1: Thinks for 25 seconds about how to respond without being socially awkward. Claude: Codes up a react app of a hand waving back to you and posts it to artifacts.

Takeaway 6: Models might need more training samples to learn to utilize larger context window sizes. We found that the model with a context window size of 8K performed better than the model with 4K, as expected. However, we observed performance was better under 8K than 16K.

Takeaway 2: SFT initialization matters: high-quality, emergent long CoT patterns from a large model (QwQ-32B) lead to significantly better generalization and RL gains compared with constructed long CoT patterns from an action prompting framework.

Don't underestimate the benefits of high-quality curated data. Turns out this is also effective in achieving complex reasoning in LLMs. With just 817 curated training samples, LIMO achieves 57.1% accuracy on the highly challenging AIME benchmark and 94.8% on MATH. Great quote from the paper: "In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning capabilities can emerge through minimal but precisely orchestrated demonstrations of cognitive processes." LIMO is based on two key factors: (1) leveraging rich mathematical knowledge already encoded in pre-trained models, and (2) using high-quality reasoning chains that demonstrate optimal problem-solving processes. This "Less-Is-More Reasoning Hypothesis" suggests that when models have strong foundational knowledge from pre-training, complex reasoning capabilities can emerge through minimal but precisely crafted demonstrations. This is more evidence that a good strong foundational pretrained model can lead to impressive results downstream. The results show significant improvements across 10 diverse benchmarks, with LIMO demonstrating exceptional out-of-distribution generalization and outperforming models trained on 100x more data.