It’s been a dream of mine since I started in ML to see autonomous agents conduct research independently and discover novel ideas! 💡 Today we take a large step towards making this a reality.
We introduce *The AI Scientist*, led together with
@_chris_lu_
and
@RobertTLange
. [1/N]
Introducing The AI Scientist: The world’s first AI system for automating scientific research and open-ended discovery!
From ideation, writing code, running experiments and summarizing results, to writing entire papers and conducting peer-review, The AI
RL agents🤖need a lot of data, which they usually need to gather themselves. But does that data need to be real? Enter *Synthetic Experience Replay*, leveraging recent advances in
#GenerativeAI
in order to vastly upsample⬆️ an agent’s training data!
[1/N]
🚨 Model-based methods for offline RL aren’t working for the reasons you think! 🚨
In our new work, led by
@anyaasims
, we uncover a hidden “edge-of-reach” pathology which we show is the actual reason why offline MBRL methods work or fail!
Let's dive in! 🧵
[1/N]
I am extremely excited to share Intelligent Go-Explore, presenting a robust exploration framework for foundation model agents! 🤖
It was a delight to work with
@shengranhu
and
@jeffclune
on this!
📜 Paper:
🌐 Website and Code:
Excited to introduce Intelligent Go-Explore: Foundation model (FM) agents have the potential to be invaluable, but struggle to learn hard-exploration tasks!
Our new algorithm drastically improves their exploration abilities via Go-Explore + FM intelligence. Led by
@cong_ml
🧵1/
Super excited to share that I'll be starting as a postdoc
@UBC_CS
with
@jeffclune
this January working on advancing open-endedness with large language/multimodal models and deep RL! 🤩
I'll be at NeurIPS next week and would love to discuss on any of those topics, I'll also be...
Offline RL offers tremendous potential for training agents from large pre-collected datasets. However, the majority of work focuses on the proprioceptive setting. In this work we release the first public benchmark for continuous control using *visual observations*, V-D4RL. [1/N]
We've now released code for this project at ! We think the potential of synthetic data for sample efficiency and robustness is huge and can't wait to see what people do with it!
In other news, we've extended the paper with pixel-based experiments... [1/2]
RL agents🤖need a lot of data, which they usually need to gather themselves. But does that data need to be real? Enter *Synthetic Experience Replay*, leveraging recent advances in
#GenerativeAI
in order to vastly upsample⬆️ an agent’s training data!
[1/N]
I love this visualisation of where we were at the start of the project (no latex, only markdown, only a few experiments). Our current version of The AI Scientist is the worst it will ever be. 🚀🚀🚀
Although The AI Scientist still makes basic mistakes, its performance will only improve from here. We see this work as being similar to early developments in GenAI, where basic mistakes in image generation were quickly overcome. Our initial manuscripts looked like this:
Thank you so much - I was so incredibly fortunate to have spent these years in Oxford under your and
@maosbot
's supervision, and will always treasure the fun discussions and lessons learned throughout!
Congratulations
@cong_ml
on defending his DPhil dissertation! Excellent work through out the past few years, and thanks to examiners
@j_foerst
@_rockt
!
Delighted that V-D4RL has been accepted at TMLR! Our benchmark and algorithms are the perfect way to start studying offline RL from pixels.
As performance in proprioceptive envs saturate, it’s increasingly necessary to look further! 🧐 Here are some notable uses so far…
[1/N]
RL agents🤖need a lot of data, which they usually need to gather themselves. But does that data need to be real? Enter *Synthetic Experience Replay*, leveraging recent advances in
#GenerativeAI
in order to vastly upsample⬆️ an agent’s training data!
[1/N]
Will be presenting our spotlight at Reincarnating RL
@iclr_conf
on generating synthetic data for RL with diffusion models at 10:40AM tomorrow!
If you can't make it, here's the pre-recorded talk:
Paper:
#ICLR2023
#GenerativeAI
Super excited to share our new work led by
@JacksonMattT
showing that policy guidance + trajectory diffusion models produce extremely strong RL training data! 💥💥
As an added bonus, our code comes with JAX implementations of offline RL algorithms and diffusion upsampling! 🚀
🎮 Introducing the new and improved Policy-Guided Diffusion!
Vastly more accurate trajectory generation than autoregressive models, with strong gains in offline RL performance!
Plus a ton of new theory and results since our NeurIPS workshop paper...
Check it out ⤵️
No better time to start on offline RL from pixels! V-D4RL is now on
@huggingface
at
💥 New D4RL-style visual datasets!
💥 Competitive baselines based on Dreamer and DrQ!
💥 A set of exciting open problems!
Thanks
@Thom_Wolf
for the idea 😻
If you’re attending
#AIIDE22
and are interested in efficient, scalable, and simple-to-implement game testing, come to the
#EXAG2022
workshop where I’ll be presenting our paper on Go-Explore for automated reachability testing at 11:20AM PDT!
Paper📜:
Come chat to us at the
#NeurIPS2023
Robot Learning Workshop in Hall B2 about policy-guided diffusion!
Super exciting work showing that guided diffusion enables long-sequence on-policy synthetic data for training agents! 🚀🚀
Come check out a sneak peek of our work **Policy-Guided Diffusion** today at the NeurIPS Workshop on Robot Learning!
Using offline data, we generate entire trajectories that are:
✅ On-policy,
✅ Without compounding error,
✅ Without model pessimism!
Come chat to us now at the
#icml
DARL workshop about simple baselines and the first public benchmark for offline continuous control from pixels! In Hall G 🥳
Magency: my project for the
@craft_ventures
agents hackathon 🤖👇
There is no good way to control AI agents on a mobile phone. There should be an app for that!
Magency is a mobile app that lets you "make a wish", aka just say what you want into one text box and see a feed of
Delighted that research from our lab was featured in Science News! Great read about harnessing large language models to create open-ended learning systems!
In many realistic imitation learning settings, we often have differences in observations between experts and imitators. E.g. when experts have privileged information.
Excited to share our new work towards a principled Bayesian solution to resolving such imitation gaps! 😍
Demonstrating the desired behavior is often easier than defining a good reward function.
However, when the demonstrator observes the world differently than the imitator, imitation learning can fail.
➡️In our new pre-print, we propose a Bayesian solution to this imitation gap.
Excited to share our ICLR spotlight on revisiting🤔 design choices in offline MBRL: later today!
In the meantime, check out our light-hearted video introducing the paper: 😀
@philipjohnball
@jparkerholder
@maosbot
, Steve Roberts
Model-based approaches have recently shown SOTA performance in the offline RL setting, typically by penalizing regions with dynamics uncertainty. But how well are current methods actually doing this? 1/
Extremely excited to share our new work led by
@GunshiGupta
and
@KarmeshYadav
showing that pretrained diffusion models provide powerful vision-language representations for control tasks that drive efficiency and generalization!
All code open-sourced at:
Excited to be giving a contributed oral talk tomorrow at the
#GenAI4DM
Workshop at
#ICLR2024
about our latest work harnessing pre-trained diffusion models as vision-language representations learners that excel across a wide variety of control tasks!
Details in 🧵below!
Come chat to us
@jparkerholder
@philipjohnball
at the
#ICLR
SSL-RL Workshop about generalisation to environments with changed dynamics from offline data on a single environment with Augmented World Models!
Gathertown Link:
Paper:
We are excited about scaling this work to more settings! This is joint work with awesome co-authors:
@philipjohnball
,
@jparkerholder
🥰. Come chat with us at the Reincarnating RL Workshop at
@iclr_conf
or get in touch!
Paper:
[7/N]
Super excited by our recent work massively expanding the scope of PBT methods in RL!
💥 Joint adaptation of architecture and hyperparameters
💥 Treating the *whole* RL hyperparameter space with trust-region BO
💥 Massive improvements on the prior PBT baselines, all code online!
(1/7) Population Based Training (PBT) has been shown to be highly effective for tuning hyperparameters (HPs) for deep RL. Now with the advent of massively parallel simulators, there has never been a better time to use these methods! However, PBT has a couple of key problems…
Really excited to share our recent work with
@philipjohnball
,
@jparkerholder
and Steve Roberts on dynamics generalisation from data from a single offline RL environment! To appear as a spotlight in the
#ICLR2021
SSL-RL Workshop 😃
The case for offline RL is clear: we often have access to real world data in settings where it is expensive (and potentially even dangerous) to collect new experience. But what happens if this offline data doesn’t perfectly match the test environment? [1/8]
Come see us tomorrow at the contributed orals at the
#GenAI4DM
workshop, talking about leveraging text-to-image diffusion models as vision-language representation learners for control!
#ICLR2024
Come chat to us now @ D5, RL4RL workshop: about our new work revisiting uncertainty quantification in offline MBRL and showcasing new SOTA results on D4RL MuJoCo 😻 Paper:
@philipjohnball
@jparkerholder
@maosbot
, Stephen Roberts
WE ARE STARTING IN 15 MIN
We have a great list of papers and guests!
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery - first author
@cong_ml
will be presenting!
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model
Check out our recent work revisiting design choices in offline model-based reinforcement learning! 🤔 arXiv: With fantastic collaborators:
@philipjohnball
@jparkerholder
@maosbot
Stephen Roberts!
Model-based approaches have recently shown SOTA performance in the offline RL setting, typically by penalizing regions with dynamics uncertainty. But how well are current methods actually doing this? 1/
(cont.) with data generated in latent space! Our experiments show a solid performance gain out of the box on the standard *V-D4RL* datasets with lots more untapped potential!
Check it out here:
[2/2]
Time to retire
@SchmidhuberAI
?
📹:
Jokes aside - this project has been soo much fun!
@_chris_lu_
and
@cong_ml
made this one of the best colabs I had so far 🥰
Our algorithm is conceptually simple and is compatible with *any RL algorithm* utilizing experience replay! Synthetic data generated by a diffusion model may simply be added to the replay buffer and trained on as if it was real experience.
[2/N]
I'm also extremely grateful to
@j_foerst
and
@_rockt
for putting me through my paces and an incredibly informative discussion! Thanks as well to all my collaborators and mentors who guided me along the way, as well as my friends and family! 🥰 Stay tuned for what's next!!
Agents trained with synthetic data perform as well as agents trained with much more real data! With *no algorithmic changes needed at all*, simple RL agents with synthetic data beat carefully designed data-efficient algorithms and prior data augmentations methods.
[3/N]
We are excited about what this unified perspective of offline RL might mean for future work!
To find out more, please check out our paper: ,
And code: ,
Thanks again to my amazing co-authors
@anyaasims
@yeewhye
! 🥰🥰
[N/N]
RL agents🤖need a lot of data, which they usually need to gather themselves. But does that data need to be real? Enter *Synthetic Experience Replay*, leveraging recent advances in
#GenerativeAI
in order to vastly upsample⬆️ an agent’s training data!
[1/N]
These edge-of-reach states trigger catastrophic value overestimation and a complete collapse of learning!
For example, this figure shows how the agent 🤖 completely ignores the reward function and instead just aims towards an arbitrary edge-of-reach state!
[4/N]
So why was this not possible before? It turns out that small differences in sample quality with VAEs and GANs significantly affect downstream RL performance.
[5/N]
We produce a vast archive of completed papers across both proprietary and open-weight LLMs, allowing us to for the first time get a sense of their ability to partake in the entire scientific process. [3/N]
We further analyze challenges and opportunities unique to the pixel-based setting including data with visual distractions. We see that our algorithms are robust to visual distractions but only Offline DV2 generalizes to unseen distractions. Scope for future work here!! 🧐 [5/N]
Remarkably, we find that the synthetic samples generated by our diffusion model are simultaneously *more diverse, more novel, and more accurate* to the true environment dynamics than the best data augmentation method.💥
[4/N]
To kick off progress on new methods, we include strong baselines derived from the SoTA DreamerV2 and DrQ-v2 algorithms! Concretely, we adapt DreamerV2 (
@danijarh
) to the offline setting by introducing a penalty based on mean disagreement, resulting in Offline DV2. [2/N]
This work was done during an awesome MAX internship over the summer hosted jointly by
@MSFTResearchCam
and
@XboxStudio
in the very lovely Cambridge. I’m super grateful to my hosts
@ralgeorgescu
and
@rookboom
, and all the friends made along the way! 🥰🥰
For context: In offline MBRL, existing methods usually assume that any issues are due to model errors. ➡️ Therefore, approaches are based around preventing model exploitation.
However, using the perfect “oracle” dynamics causes existing methods to completely fail! 📉
[2/N]
We found our base algorithms already represent a strong multitask baseline, opening the door to training generalist agents from offline data. 🤯 This could be because we can directly distinguish between different tasks instead of relying on explicit meta or multitask algos! [6/N]
💥 ML Research Opportunity for under-represented undergrads at Oxford! 💥
Would appreciate help sharing this widely! UNIQ+ is an awesome way to spend two months getting stuck into ML at great groups
@oxcsml
@CompSciOxford
See proposed projects here:
But, existing methods work?! We discuss how they inadvertently address edge-of-reach states despite their motivation from model error.
Can we directly target the true problem? Yes! We introduce RAVL, which precisely corrects edge-of-reach states using value pessimism.
[5/N]
Towards this goal, we were curious to see how our baselines scaled with more data. Interestingly, the RL methods scale far better than BC, with gains of >30% compared to 10% when we go from 100K samples to 500K! This may have implications for when we scale even further! 🚀 [7/N]
@Hoper_Tom
@jeffclune
Thank you for kindly sharing these works! We will discuss these in the updated version of the paper, and also look forward to integrating the insights from your paper into our work!
So, what is going on?
We show that there exist “edge-of-reach” states which are used in training but which the agent can never sample actions from *even with unlimited model-based data collection* (as illustrated below).
[3/N]
We adapt the algorithm DrQ-v2 (
@denisyarats
) by adding an adaptive behavioral cloning term similar to TD3+BC, resulting in DrQ+BC. We also include a CQL and BC implementation in the same codebase. [3/N]
Another key benefit we observe on some algorithms is the ability to scale up the network size and obtain better performance! RL algorithms are typically limited to training with very small networks and synthetic data could lead to the lifting of this restriction!
[6/N]
Given a starter research area (e.g. diffusion, language modeling), The AI Scientist generates novel research ideas, writes code, executes experiments, visualizes results, describes its findings by writing a full scientific paper, and then runs a review process for eval. [2/N]
We find that the model-based Offline DV2 performs best on datasets with diverse data, DrQ+BC deals well with mixed but high-reward data and BC is best at the expert datasets. CQL is also a strong option for high-reward data. [4/N]
RAVL is theoretically principled and reaches state-of-the-art on D4RL and pixel-based V-D4RL without any explicit dynamics penalty! 📈
Furthermore, these insights serve to correct and unify our understanding of offline RL across model-free and model-based approaches.
[6/N]
@jsuarez5341
@arankomatsuzaki
We found this as well in a project that sounds v close to both these efforts! See Table 3 of where we show diffusion synthetic data can help scale the network sizes in TD3 :)
Efficient online reinforcement learning with offline data ()
@philipjohnball
@ikostrikov
@smithlaura1028
showing a remarkably simple method to accelerating online pixel-based training with V-D4RL datasets!
[5/N]
@Stone_Tao
Thanks! At the moment, it's roughly 50/50 for diffusion vs. RL training. Big potential for speed-ups there though. DMC is on proprioceptive, visual transitions incoming! :)
Agent-controller representations: Principled offline rl with rich exogenous information ()
@riashatislam
@manan_tomar
learning how to handle rich amounts of irrelevant information commonly found in pixel-based datasets!
[2/N]
Behavior prior representation learning for offline reinforcement learning ()
@Hongyu
Zang
@Xin
Li learning effective state representations by behavioral cloning in V-D4RL!
[3/N]
Revisiting the Minimalist Approach to Offline Reinforcement Learning ()
@ML_is_overhyped
@vladkurenkov
integrating many simple recent algorithmic advances into the DrQ+BC baseline provided by V-D4RL!
[4/N]