🚨 Introducing “MMSci: A Multimodal Multi-Discipline Dataset for PhD-Level Scientific Comprehension” 🧐Have current multimodal LLMs achieved PhD-level intelligence across diverse scientific disciplines? Are they ready to become AI scientific assistants? 📢We announce MMSci, a multimodal, multidisciplinary dataset sourced from published articles spanning 72 disciplines from Nature Communications journals #NatureComms, to evaluate and enhance models' comprehension of Phd-level scientific knowledge. ✨Highlights of MMSci: - 72 diverse advanced scientific disciplines, including physics, chemistry, materials science, nanoscience, optics and photonics, biochemistry, energy science, ecology, climate science, ocean science, genetics, immunology, social sciences, agriculture, etc. (. - 131k articles and 742k figures directly crawled from the web instead of extracted from PDF, ensuring diversity and quality. - Heterogeneous and complex multi-panel scientific figures, including charts/graphs, schematic diagrams, macroscopic/microscopic photographs, simulated images, geographical maps, and more. - Benchmarking LMMs' understanding of scientific figures and content, across varying settings. - Visual instruction-tuning data and interleaved article and figures for LMM visual pre-training. 📊 Results and Takeaways on evaluating OSS Models, #GPT4V and #GPT4o: > OSS LMMs showed limited capability in understanding scientific figures, performing near random guesses. #GPT4V and #GPT4o also faced difficulties in challenging settings, achieving 50%-70% accuracy. > Writing relevant and concise captions for scientific figures requires conditioning on the article content, especially the full content to achieve reasonable caption. > Our constructed visual instruction-tuning data improved a 7B LLaVA-Next(v1.6) model to achieve performance comparable to GPT4V/o on our benchmark. > The interleaved article and figure data could be used for LMM pre-training to infuse scientific knowledge, showing improvement on material science tasks.

Big new review! 🟦Open Problems in Mechanistic Interpretability🟦 We bring together perspectives from ~30 top researchers to outline the current frontiers of mech interp. It highlights the open problems that we think the field should prioritize! 🧵