Achieve SOTA on multimodal, complex tasks with minimal supervision! Introducing In-Context Abstraction Learning (ICAL): a method that extracts insights from visual demos and human feedback to continually teach multimodal LLM agents 🚀🤖🧠 ICAL learns from sub-optimal demonstrations and human feedback by transforming noisy trajectories into optimized, annotated examples that improve multimodal decision-making in new domains. ICAL is the first to show state-of-the-art performance across three challenging domains: we obtain a 34% relative improvement on TEACh household robot instruction following from dialogue, a 59% relative improvement on Visual Web Arena for autonomous visual web agents, and beat few-shot GPT-4V and supervised baselines on Ego4D action forecasting.

OpenAI's announcement of Operator on Thursday was a great excuse for us to come out of stealth to show off the AI agents tech we've been building at Yutori. Which means I can now say out loud — we're hiring! Our current top hiring priorities are an awesome founding frontend engineer, followed by backend and full-stack engineers, followed by designers and AI scientists/engineers. Apply, and spread the word! @yutori_ai Link👇

Adverse Drug Reactions are among the leading causes of hospitalizations for mental health issues. LLMs have the potential to detect ADRs and provide mitigation strategies. But do LLMs align with experts? 🤔 We explore this in our work 👇🏼🧵 1/11

There’s still time to apply to work with me at Utah. If you’re interested in the intersection of language with perception and action and/or web agents, definitely apply!

Excited to share that ICAL has been accepted as a #NeurIPS2024 *spotlight*! 📄 🌟 ICAL enables VLMs to learn efficiently from noisy demos and human feedback, creating multimodal programs of thought that capture critical task dynamics and abstractions.