🌐 Are LLM agents prepared to navigate the rich diversity of cultural and social norms? 🏠 CASA tests them on real-world tasks like online shopping and social discussion forums, revealing that current agents show less than 10% awareness and over 40% norm violations. 🧠 We’re bridging this gap by combining fine-tuning on regional data with strategic prompts to create agents that better understand our world’s diversity. Read the full paper for all insights! 📑 Grateful for the incredible team at Salesforce AI Research @salesforce !

🌍Are LLMs aware of cultural and legal safety in today’s geo-diverse world? 🚀Introducing SafeWorld, our #NeurIPS2024 paper and benchmark assessing LLMs’ understanding of geo-diverse safety, based on cultural norms and policies across 50 countries and 493 regions/races. ⚖️We also propose a multi-dimensional framework for evaluating contextual appropriateness, accuracy, and comprehensiveness, revealing major gaps in current LLMs. 🧨To address this, we train SafeWorldLM using DPO, achieving SOTA performance and a 20% higher global human evaluator rating in helpfulness and harmfulness over competing models, including GPT-4o. 🔗Paper: 💻 GitHub: 🫶🏻This is a joint leading effort with @Wade_Yin9712. Also many thanks to the amazing team @steeve__huang @kaiwei_chang, and @VioletNPeng for their hard work. Check out more details and results we conclude from our paper in the thread below. 🧵