Quan Vuong
@QuanVng
Followers
1,790
Following
239
Media
19
Statuses
420
robotics research and co-founder at @Physical_int , ex- @GoogleDeepMind Perpetually trying to find a quiet place to read.
Joined January 2015
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
FEMA
• 1203388 Tweets
Liz Cheney
• 209584 Tweets
#LISAxMoonlitFloor
• 197221 Tweets
MOONLIT FLOOR OUT NOW
• 137932 Tweets
SCJN
• 128619 Tweets
The Boss
• 103770 Tweets
#GHGala5
• 100032 Tweets
Bruce
• 92338 Tweets
Mets
• 69981 Tweets
Happy Anniversary
• 59797 Tweets
Baker
• 51010 Tweets
Brewers
• 42600 Tweets
EL DESTELLO IS OUT
• 34794 Tweets
天使の日
• 33230 Tweets
Falcons
• 29013 Tweets
Mancuso
• 25832 Tweets
Halle
• 24850 Tweets
もちづきさん
• 24728 Tweets
Pete Alonso
• 19614 Tweets
Athena
• 15232 Tweets
Mike Evans
• 14530 Tweets
Bijan
• 13510 Tweets
Phillies
• 11404 Tweets
#ゴンチャのハロウィン準備中
• 10748 Tweets
Milwaukee
• 10479 Tweets
Quintana
• 10036 Tweets
Pinned Tweet
RT-X: generalist AI models lead to 50% improvement over RT-1 and 3x improvement over RT-2, our previous best models. 🔥🥳🧵
Project website:
7
143
621
🚨 Big news 🚨
Together with a set of amazing folks we decided to start a company that tackles one of the hardest and most impactful problems - Physical Intelligence
In fact, we even named our company after that: or Pi (π) for short
🧵
46
46
549
7
5
66
Large, ambitious projects in robotics are fun and really worthwhile
We should do more of them!
1
0
42
Pictures taken at RT-2 poster at
@DannyDriess
requests ; )
@YevgenChebotar
We miss you
@TianheYu
CC
@hausman_k
1
4
37
The code for optimistic actor critic is now open-sourced.
Optimistic Actor Critic, with the principle of optimism in the face of uncertainty, obtains an exploration policy by using the upper bound instead of the lower bound. Learn how OAC increases sample efficiency compared to other methods:
#NeurIPS2019
2
76
561
0
6
30
Co-training with static data improves perf on mobile tasks. Signs of generalist policies to come : )
Introducing 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀🏄 -- Hardware!
A low-cost, open-source, mobile manipulator.
One of the most high-effort projects in my past 5yrs! Not possible without co-lead
@zipengfu
and
@chelseabfinn
.
At the end, what's better than cooking yourself a meal with the 🤖🧑🍳
235
1K
5K
2
5
28
Tried mathpix to convert picture into latex and I was blown away by how well it worked 😍
9
54
154
0
6
29
Mind-blown from reading this paper: meta-gradient reinforcement learning (). A learning algorithm which learns to edit itself online during the training process at no extra data cost. Online cross-validation is so cool 🤯🤯🤯
0
13
24
@KarlPertsch
and I are giving a talk about RT-X future plans at Room 354 at 3PM, come discuss with us!
#NeurIPS23
1
1
23
1. Optimistic Actor Critic (neurips 2019 spotlight)
Existing tricks to stabilize training leads to pessimistic exploration. We introduce optimistic exploration and obtain sample efficiency gains!
Paper:
1
0
18
We are open-sourcing the X-Embodiment datasets in an unified format to enable even more interesting research in this area.
1
1
17
The era of OSS generalist robot action models has begun?
3 mo. ago we released the Open X-Embodiment dataset, today we’re doing the next step:
Introducing Octo 🐙, a generalist robot policy, trained on 800k robot trajectories, stronger than RT-1X, flexible observation + action spaces, fully open source!
💻:
/🧵
10
90
373
0
2
15
To evaluate the RT-2-X model, we host the model in the cloud and query the model over the internet to run evaluation at Stanford and Berkeley. A glimpse into the robot cloud API future!
2
4
14
Excited to share 2 accepted papers at NeurIPS on RL, with 1 spotlight!
3
0
14
When great works like this come out, we all win!
Proud to announce Dobb·E: the next step in home robot system that I was working on for the past 3 years.
We have visited 10 homes, learned 100+ tasks, and we are just getting started!
And we fully open-sourced it all, hardware, models, and software: 🧵
20
100
484
0
3
12
Positive transfer between manipulation and navigation!
Sign of X-embodied policies to come : )
Cross-embodied robot policies hold the promise of one policy to control all robots. But how far does transfer go? In new work, we study positive transfer between *manipulation* & *navigation* and show that nav data helps manipulation, and vice versa!
🧵 👇
1
45
181
0
0
10
New work on domain randomization!
Joint work with
@sharadvikram
, Dr. Hao Su, Dr. Sean Gao and my dear advisor
@hiskov
Paper:
Code:
1
3
10
To evaluate the RT-1-X model, we sent the model checkpoints to 5 different academic labs and ran evaluation using existing robot infrastructure and control stack without any modifications. 🙀
We did not standardize the control stack across the 5 different labs.
1
1
9
The project is a collaboration between 173 researchers from 34 different research labs. We pooled together data to create one of a kind data sets, containing 22 embodiments.
1
1
9
Had a blast working with
@priyasun_
on RT-Sketch! Such a fun and creative project!
Check out Priya's thread below!
We can tell our robots what we want them to do, but language can be underspecified. Goal images are worth 1,000 words, but can be overspecified.
Hand-drawn sketches are a happy medium for communicating goals to robots!
🤖✏️Introducing RT-Sketch:
🧵1/11
6
49
270
0
0
9
Blog post by Google DeepMind
0
1
8
After I started working with Master students, I have a new found respect and appreciation for what it is my PhD advisors do :)
0
0
8
Data analysis thread by
@KarlPertsch
Very excited to release the Open X-Embodiment Dataset today — the largest robot dataset to date with 1M+ trajectories! Robotics needs more data & this is a big step!
There’s lots to unpack here, so let’s do a deep dive into the dataset!
🧵1/15
8
90
451
1
3
8
Real2Sim2Real for 6DOF grasping in clutter using neural surface reconstruction!
Paper:
Video:
1
4
8
Modeling wise, we made minimal changes to RT-1 and RT-2 and were surprised that we obtained performance improvement out of the box.
We refer to the RT-1 and RT-2 model trained on the X-Embodiment dataset as RT-1-X and RT-2-X.
1
1
8
I was really surprised how well classification loss worked!
Check out Aviral's thread below.
Super simple code change to get value-based deep RL scale *much* better w/ big models across the board on Atari games, robotic manipulation w/ transformers, LLM + text games, & even Chess!
Just use classification loss (i.e., cross entropy), not MSE!!
🧵⬇️
3
42
266
0
0
6
The lab logos indicate the physical location of real robot evaluation, and the robot pictures indicate the embodiment used for the evaluation.
1
1
7
For any inquiries, please email open-x-embodiment
@googlegroups
.com
Thank you!
RT-X: generalist AI models lead to 50% improvement over RT-1 and 3x improvement over RT-2, our previous best models. 🔥🥳🧵
Project website:
7
143
621
0
0
7
First day at work!
I’m so incredibly excited to spend the summer doing machine learning research at Microsoft Research Cambridge! Wooo ❤️❤️❤️
1
0
6
The best is yet to come 😀
It's been a few days since the RT-X release, and one of the most gratifying things to me in the reaction is the recognition of how much this was a team effort -- a large portion of the robotic learning community coming together to do something bigger than any one lab could do.
6
15
189
0
0
6
👀 👀 👀
Many researchers have asked us about sharing our RT dataset and making it easier to participate in large-scale robot learning research.
We're working on it and we'll have some updates on this soon! 👀
6
11
122
0
0
5
Cross-painting allows for zero-shot generalization of end2end policies to unseen robot arms!
Check out Lawrence's thread below!
Introducing Mirage: Zero-shot transfer of visuomotor policies to unseen robot embodiments 🤖
With Mirage, you can train a policy on one robot and deploy it on a different one that it has never seen, with no additional data or training! 🧵👇 (1/8)
🌐
4
21
88
0
0
5
I love the optimism
@adcock_brett
The timeline split of AI vs Robot Hardware has changed
the last 90 days i’ve witnessed industry leading AI in our lab running on humanoid hardware, and frankly it’s blown me away
i’m watching robots performing complex tasks entirely with neural nets. AI trained tasks that i
148
381
2K
0
0
4
@xf1280
@DrJimFan
@scott_e_reed
Thanks Fei!
Please note 3Hz is the system-level latency, e.g. including camera and communication overhead.
The neural network itself runs much faster (Table 13 on page 30 fyi)
0
0
4
2. Pre-training as Batch Meta Reinforcement Learning with tiMe
We introduce a pre-training method for RL that only uses observational data and NO environment interaction during meta-train
It generalizes zero-shot to unseen MDP. Important to allow for scalable data collection.
1
0
4
👀 👀 👀
So far, there have been some remarkable large-scale robotic learning results, datasets, and milestones this year. But we have something pretty big coming out tomorrow. So big that we needed a globe to visualize its scale😉
24
48
772
0
0
4
Online technical meeting is so much more draining compared to in-person …
0
0
2
1. Using completely offline data to accelerate training on unseen tasks (up to 70%, even on Humanoid!)
Arxiv:
1
0
3
@nguyentienvu
Reminds me of an email that starts with “It is our pleasure to inform you that your grant application has been rejected...” true story 😀😀😀
0
0
3
Congratulations!
📢Thrilled to announce sudoAI (
@sudoAI_
), founded by a group of leading AI talents and me!🚀
We are dedicated to revolutionizing digital & physical realms by crafting interactive AI-generated 3D environments!
Join our 3D Gen AI model waitlist today!
👉
14
101
427
0
0
3
Prefrontal cortex as a meta-reinforcement learning system. So wild.
0
0
3
With Kamil Ciosek, Robert Loftin, Katja Hofmann of MSR Cambridge.
My contribution was done during my internship, from which I grew a whole lot!
If you want a non-trivial probability of producing a spotlight, apply here : )
0
1
3
3. Streamlined Off-Policy Learning
How does max ent helps RL?
We demonstrate that Soft Actor Critic is solving the bounded nature of the action space.
1
1
3
SOTA grasping network fails catastrophically when transferring to new robot morphologies because the network overfits to the geometry of the gripper.
Our approach recovers >90% grasping performance, without training on any real world grasping data.
1
0
3
2. An efficient, simple and theoretically motivated method for safe RL! The techniques should be applicable to any optimization problem where the objective is convex in the output of the NN!
Arxiv:
1
0
2
“This suggests that latent knowledge regarding future discoveries is to a large extent embedded in past publications.”
What do you think?
"unsupervised method can recommend...for applications...several years before their discovery."
@AndrewYNg
@ylecun
@drfeifei
@etzioni
@demishassabis
@ShaneLegg
@mustafasuleymn
@GaryMarcus
@frossi_t
@rodneyabrooks
#machinelearning
#AI
6
50
169
0
0
2
So wild
"A neural network trained to predict future video
frames mimics critical properties of biological
neuronal responses and perception"
@neurobongo
0
3
7
0
0
2
But can we get automatic batching like jax or dynet pretty please ? Good for ensembles, set NN, and much less mental overhead overall.
PyTorch 1.3 includes support for model deployment to mobile devices, quantization, & front-end improvements, like the ability to name tensors. New tools & libraries are also launching for improved model interpretability & multimodal development. Read more:
7
223
635
0
0
2
I had a great time interning in Katja's team this summer. Grew a lot as a researcher and a person! Highly recommend!
We have an exciting new internship opportunity in my team
@MSFTResearchCam
- focusing on Reinforcement Learning for Game Intelligence - - apply now!
1
32
84
0
0
2
@micoolcho
@xiao_ted
@pannag_
@hausman_k
@chelseabfinn
@frodobots
Please email open-x-embodiment
@googlegroups
.com instead of DM to help us track communication easier.
Thank you!
0
1
2
Given RGBD observations of a table top scene, we:
1. reconstruct the geometry of the objects in the scene
2. place the reconstructions in a simulated environment (without needing pose estimation at all)
3. use the reconstructions to train or fine-tune grasping networks
1
0
2
@TacoCohen
@shaneguML
Where is the data going to come from? Will the model be transformer-based or rl-based or a bit of both?
2
0
2
0
0
2
@ericjang11
@TacoCohen
@shaneguML
But in all seriousness, I mean "where does the interaction data come from?"
1
0
1
Played escape the room ytd. Must be how it felt to be an RL agent, forced to generalize to an unseen MDP with spare reward function, guided by a learnt intrinsic reward function 🧐
0
0
2
Ratatouille soundtrack is so delightful. "Is it soup yet?" is especially intense 😂😂😂
0
0
2
Accepted at the Learning Legged Locomotion Workshop at International Conference on Robotics and Automation 2019 (ICRA)! So sad I can’t make it 😥
New work on domain randomization!
Joint work with
@sharadvikram
, Dr. Hao Su, Dr. Sean Gao and my dear advisor
@hiskov
Paper:
Code:
1
3
10
1
1
2
@ericjang11
@TacoCohen
@shaneguML
But … but my primary school teacher told me Wikipedia is not a credible info source 😜
1
0
2
Can GPT-3 write my rebuttal for me? 😆😆😆 very poss to prime it with successful rebuttals in the past Mhm
0
0
2
Interesting that they allow human to tele-operate the arms instead of letting the robot autonomously propose goals.
Can this be a design choice to maintain safety during training?
Excited to share our new work on learning from play!
We show a single agent, after self-supervising on 3 hours of play data, can generalize to 18 zero-shot manipulation tasks with 85% success.
interactive paper:
1/
1
73
192
1
0
1
Paper:
With Shuang Liu, Minghua Liu, Kamil Ciosek, Hao Su, Henrik Christensen of UCSD and MSR Cambridge.
0
0
1
0
0
1
Opening a terminal inside vs code automatically opens a shell on the remote machine AND activates the current conda environment 🤯🤯
1
0
1
@weights_biases
can we please have a way to compute correlation between different logged metrics?
Would be very useful for debugging. Thank you!
1
0
1
Major props to
@sharadvikram
, whose software engineering skills and research advices made my life so much easier in this project! 🙏🙏🙏
1
0
1
Heading home soon from NeurIPS. Felt as invigorated to do research as ever.
0
0
1