Evaluating on our ScienceAgentBench (Coding tasks in Bioinformatics/Chemistry/Geo info science/Cognitive science) just got much easier and faster! Check out our update on containerized evaluation: (1) Task environments are set up in independent docker containers, which eliminates potential package conflicts among different tasks and allows us to remove pip-tools, a major factor of slow evaluation previously. (2) Users can now evaluate their agents using a single bash command and no longer need to set up their own conda environments. (3) With multi-threading, programs for each task can be configured and executed in parallel, reducing the evaluation time to only 20-30 minutes for all 102 tasks. Great efforts led by awesome @YifeiLiPKU and @BotaoYu24 @osunlp!

Very excited to learn that our 2023 paper, "G2Retro as a two-step graph generative models for retrosynthesis prediction ()," (led by @ziqiChen123 and @ningx005) has been selected into Nature's special collection, "Nobel Prize in Physics 2024, This collection highlights high-impact research, reviews, and opinion articles selected from all of Nature's participating journals, celebrating “the direct contributions by the [Nobel Prize] awardees and the advances they have inspired.” Building on the momentum, earlier this year, we released SMolInstruct (a large-scale, high-quality instruction tuning dataset for small molecules) and LlaSMol (state-of-the-art instruction following LLMs for a variety of chemistry tasks), led by awesome @BotaoYu24 @osunlp. Recently, we released ChemAgent (also led by @BotaoYu24), a state-of-the-art chemistry agent equipped with 29 tools for both specialized tasks and general questions in chemistry, built on top of the pioneering work, ChemCrow. Around the same time, we released ScienceAgentBench (led by our great @RonZiruChen), a new coding benchmark for scientific tasks, to rigorously evaluate agents in assisting scientists with programming for data-driven scientific discovery. ScienceAgentBench features 102 *real-world* tasks extracted from 44 peer-reviewed publications across 4 scientific disciplines (Bioinformatics, Computational Chemistry, Geographical Information Science, Psychology & Cognitive Neuroscience). Our latest result shows #o1-preview with self-debug can achieve 42.2% task success rate (10% more than other agents) while costing $0.6-0.7 per task (>10 times more than others). See details about each project in this thread: