Rice CS/ECE is hiring for tenure-track faculty position in Computer System. Feel free to check out this post for more detail.

The paper says, LLMs approach arithmetic as position-based pattern matchers, rather than computational engines. So LLMs learn arithmetic through symbolic abstraction instead of mathematical calculations Original Problem 🔍: LLMs struggle with basic arithmetic despite excelling in complex math problems. Previous research focused on identifying model components responsible for arithmetic learning but failed to explain why advanced models still struggle with certain arithmetic tasks. ----- Solution in this Paper 🛠️: • Introduced a subgroup-level framework to analyze how LLMs handle arithmetic learning • Broke down arithmetic tasks into two aspects: 👉 Subgroup complexity: measured through domain space size, label space entropy, and subgroup quality 👉 Subgroup selection: how LLMs choose input-output token mappings during training • Tested four different multiplication methods to investigate if LLMs use partial products • Analyzed position-level accuracy across different training sizes ----- Key Insights 💡: • LLMs don't perform actual calculations but function as pure symbolic learners • Label space entropy is crucial for task complexity measurement • LLMs show U-shaped accuracy curve - high accuracy (95%+) at first/last digits, low (<10%) in middle positions • Models select subgroups following an easy-to-hard paradigm during learning • Explicit training on partial products doesn't improve multiplication performance ----- Results 📊: • Models improved in identifying partial products after training but failed to leverage them for calculations • When subgroup complexity remained fixed, LLMs treated different arithmetic operations similarly • Reducing entropy through modular operations improved accuracy • Position-level accuracy followed consistent U-shaped pattern across different training sizes • Both Gemma-2-2B and Llama-3.1-8B showed similar symbolic learning patterns