Vikash Kumar
@Vikashplus
Followers
3,779
Following
438
Media
222
Statuses
947
Studying intelligent embodied behaviors. Ad. Prof. @CMU_Robotic | Sr. research scientist at @AIatMeta @GoogleAI @OpenAI | @berkeley_ai @UWcse #MuJoCo
Pittsburgh
Joined February 2016
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
松野さん
• 109173 Tweets
松野太紀さん
• 90588 Tweets
RENO12 WITH MILKLOVE
• 81866 Tweets
金田一少年の事件簿
• 57889 Tweets
犬夜叉の鋼牙
• 52087 Tweets
万丈目さん
• 49169 Tweets
スポンジボブ
• 45322 Tweets
#映画よめぼく
• 38158 Tweets
Burberry x Pond
• 37624 Tweets
ANDAM JURY ROSÉ
• 27928 Tweets
#初星学園HR
• 23096 Tweets
كوريا
• 16721 Tweets
Patrick Underwater🫧
• 15874 Tweets
Razem
• 15810 Tweets
アナザーアイドル
• 13063 Tweets
一ちゃん
• 11234 Tweets
اليابان
• 10786 Tweets
Pinned Tweet
#𝗥𝗼𝗯𝗼𝗔𝗴𝗲𝗻𝘁 -- A universal multi-task agent on a data-budget
💪 with 12 non-trivial skills
💪 can generalize them across 38 tasks
💪& 100s of novel scenarios!
🌐
w/
@mangahomanga
@jdvakil
@m0hitsharma
, Abhinav Gupta,
@shubhtuls
4
67
248
Lack of scale & diversity in robot datasets is demanding a change towards scalable alternatives- LLMs, Sim2Real, etc.
GenAug presents a powerful recipe for using text2image generative models to demonstrate widespread generalization of robot behaviors to novel scenarios.
🧵(1/N)
2
45
194
📢#𝗥𝗼𝗯𝗼𝗛𝗶𝘃𝗲 - a unified robot learning framework
✅Designed for genralizn first robot-learning era
✅Diverse (500 envs, 8 domain)
✅Single flag for Sim<>Real
✅TeleOper Support
✅Multi-(Skill x Task) realworld dataset
✅pip install robohive
🧵👇
3
34
195
📢 Announcing a breakthrough in science robotics
@SciRobotics
- 𝙑𝙞𝙨𝙪𝙖𝙡 𝘿𝙚𝙭𝙩𝙚𝙧𝙞𝙩𝙮
🏳️🌈 any object
🌈 any rotation
📉 a low-cost hand (D'Claw)
📷single camera
A single policy capable of in-hand reorientation of
novel & complex objects
(thread👇)
6
30
182
#MuJoCo
3.1.4 drops💧
Lots of new features -
- Gravity compensation
- Non-linear least squares (ala SysID✅)
- MJX features (yup, still no tendons🫤)
- many more...
3
24
179
Just wait until you realize that this is full-blown physics with contacts, not just a visual rendering!
💪
#MuJoCo
3.0 is packed with features
👉 XLA: Accelerated physics with JAX
👉 Non-convex collisions
👉Deformable bodies
Give it a try -
1
15
167
Human Dexterity is the epitome of motor control. My PhD goal was to study this grand challenge.
Amazes me to-date, we found 1st behavioral prior for human dexterity
#MyoDex
. It's a step change🪜in our understanding, & ability to synthesize, physiological motor control.
#proud
2
26
162
New paper - RelayPL
Takeaway: Goal conditioned hierarchical policy when infused with unstructured play data and a relabelling trick can solve REALLY LONG horizon tasks!
Website:
With A. Gupta,
@coreylynch
,
@svlevine
,
@hausman_k
1
28
122
It has been a long roller coaster journey that started in 2008 when I was an intern under Emo Todorov. 12 years and now it's in the hands of the open source community. Excited for the next phase ...
Big congratulations to team MuJoCo and
@DeepMind
We’ve acquired the MuJoCo physics simulator () and are making it free for all, to support research everywhere. MuJoCo is a fast, powerful, easy-to-use, and soon to be open-source simulation tool, designed for robotics research:
85
2K
6K
0
12
121
What do robot-learning & horse-racing have in common?
All candidates can never be simultaneously compared in identical real world conditions. Yet horse-racing has a leader-board & robot-learning doesn’t!
Presenting Ranking Based Robotics Benchmarks
1/N
1
31
119
Introducing a game-theoretic framework MBRL-GAME that (a) unifies existing MBRL methods (b) derives algorithms that are stable & performant (c) delivers 𝙎𝙊𝙏𝘼 results on OpenAI gym, ROBEL, and hand manipulation suite
Web:
Paper:
2
22
103
ATLAS is such a show-off 😎
Had a phenomenal visit to
@BostonDynamics
discussing "Foundation Models for Low Level Motor Control". (Recording coming soon)
Thanks
@KuindersmaScott
for the invite
#bigfan
. He is onto something cool, keep your eyes peeled and fingers crossed.
4
16
100
📢Major update to
#RoboHive
v0.7🔥
"Unified Robot Learning Framework"
Seamlessly work across
#Gym
&
#Gymnasium
w multiple teleop interfaces(sim & hardware) on latest
#MuJoCo
🌐
🏗️ pip install robohive
1
15
90
𝐑𝐑𝐋: Surprisingly simple method at the intersection of representation, imitation, and reinforcement learning that leverages features from standard pre-trained image classification models as representations to deliver complex contact-rich behaviors.
1/N
1
10
86
Woohoo,
#R3M
won the best paper award at
#ICRA
Scaling robot learning workshop. ( )
@SurajNair_1
,
@aravindr93
,
@chelseabfinn
, Abhinav Gupta
6
7
85
Learned controller (white) vs classical controller (grey) compete head to head @
#corl2022
PS: both are being tele-oprerated
2
15
84
Can we solve robotics using off-domain, non-robotics datasets?
I'm bullish on➡️ LLMs for high-level plans 🔄 human videos for low-level robot skills.
📢𝑯𝑶𝑷𝑴𝑨𝑵 -- zero-shot manipulation in the wild from human videos!
with
@mangahomanga
, Abhinav Gupta,
@shubhtuls
🧵👇
2
7
78
Why does Reinforcement Learning(RL) struggle with high-dim control problems?➡️Exploration
Announcing the biggest speed-up in RL-algos in recent history 𝙎𝙮𝙣𝙚𝙧𝙜𝙞𝙨𝙩𝙞𝙘 𝘼𝙘𝙩𝙞𝙤𝙣 𝙋𝙖𝙩𝙝𝙬𝙖𝙮𝙨!
✅compatible with all RL-libs
😀fully automatic
❌no demos required
1
10
77
We are building something unimaginable 🧐
Hiring
#ML
, 3D
#vision
experts and tech-artists at all levels. Will love to help out. DMs open
Between Apple SPG shutdown, Cruise layoffs, and shakeup at Google Robotics there are >1,000 top notch AI & Robotic engineers on the market in the Bay Area 🙌
Bay area continues to be top place imo for robotics - 5yrs ago i moved from NYC here for this very reason
19
15
367
4
6
78
Human possess a remarkable ability to perform a broad spectrum of tasks with varying degrees of precision. Can we imparting similar levels of versatility to robots merely by watching human videos?
📢 H2R - an agent capable of common coarse-manipulation skills
1/N🧵
2
12
68
Often overlooked in AI, morphological representations are one of the strongest form of prior behind ALL intelligent being
- acquired over generations
- has no replacement
- more fundamental+critical than acquired representations (world models, concepts, etc)
ダチョウから着想を得た脚部クラッチにより、最小限の制御でエネルギー効率の高い歩行を実現した鳥ボット
#BirdBot
#robot
#robotics
#Biorobotics
#BioRob
#biomimicry
#バイオミミクリー #生物模倣 #ダチョウロボット
9
407
1K
1
10
60
The uniqueness of human intellect lies at the juncture of 1. cognitive decision making, &
2. musculoskeletal motor-control to express them.
Introducing
#MyoSuite
-- a framework to investigate these two facets of intelligence.
1
8
65
Presenting the newest member of the
#ROBEL
family:
⭐️D'Manus⭐️- a learning platform for tactile aware manipulation
📉 Low cost ($3500)
👌10 DoF prehensile hand
👋 Surface tactile sensing (ReSkin)
💪 Robust (>10k operational hours)
✅ Open-source
1/n
3
11
62
Reproducibility in robotics is desired, but extremely challenging, especially with hardware results.
Our effort in making
#ALOHA
reproducible is paying off. Multiple folks have been able to reproduce and extend it to new systems.
A thread 👇
2
10
56
🌈𝙀𝙭𝙥𝙖𝙣𝙙𝙞𝙣𝙜 𝙁𝙧𝙤𝙣𝙩𝙞𝙚𝙧𝙨 𝙤𝙛 #𝙎𝙞𝙢2𝙍𝙚𝙖𝙡🌈
📢
#ICRA
workshop deliberating a transformative paradigm that has pushed many frontiers.
🔎 will it keep scaling/delivering?
🔎 open/missed opportunities?
📩Accepting submissions. See you at
@ieee_ras_icra
1
12
55
The official RL library of
#PyTorch
is finally out📢📣
Thrilled to have played a small role. Take it for a whirl 🌪️ Looking forward to your valuable feedback.
TorchRL has its paper!
With Albert Bou,
@MatteoBettini98
, Sebastian Dittert,
@Vikashplus
,
@shagunsodhani
, Xiaomeng Yang,
@gdefabritiis
and
@VincentMoens
1
18
63
1
8
52
📢Announcing:
#MyoChallenge
@
#NeurIPS2022
🎯Goal: Physiological Dexterity
💡Task: Develop controllers for physiologically realistic musculoskeletal
#MyoHand
to solve complex dexterous manipulation tasks
🎲🔃 die reorientation
⚪🔄 simultaneous rotation of two Baoding balls
1/3
3
9
52
🎉 Quite pleased to share our
#GenerativeAI
efforts in robotics-
#𝗚𝗲𝗻𝗔𝘂𝗴 has been nominated for the 𝗕𝗲𝘀𝘁 𝗦𝘆𝘀𝘁𝗲𝗺 𝗣𝗮𝗽𝗲𝗿 𝗔𝘄𝗮𝗿𝗱🏅of
#RSS2023
!
w-
@ZoeyC17
@shokiami
@abhishekunique7
at
@uwcse
,
@MetaAI
Lack of scale & diversity in robot datasets is demanding a change towards scalable alternatives- LLMs, Sim2Real, etc.
GenAug presents a powerful recipe for using text2image generative models to demonstrate widespread generalization of robot behaviors to novel scenarios.
🧵(1/N)
2
45
194
3
2
49
How robot videos should be - uncut and at 1x
Kudos to the
@1x_tech
team for this amazing feat.
3
3
48
While others r moving to Imitation Learning,
#OG
of model-based control
@BostonDynamics
deploys new SPOT capabilities with RL!
What's exciting? ideological camps in robotics are breaking down. The conversation is about mixing, not choosing between different approaches...
(1/2)
2
4
47
Rethinking dexterous manipulation from the first principles-
✅Algorithm (MTRF) that can learn behaviors autonomously without *any* human intervention
✅Paradigm that can learn multiple tasks simultaneously
✅New D'Hand that is robust to over ~2000 hours of on-hardware training!!
After over a year of development, we're finally releasing our work on real-world dexterous manipulation: MTRF. MTRF learns complex dexterous manipulation skills *directly in the real world* via continuous and fully autonomous trial-and-error learning.
Thread below ->
5
53
323
1
9
46
Early years is all about building representations (Vision, motor, world, morphology) strong enough to simplify decision making and execution.
My 2023 realization is representation>control
Lots of roboticists talk about how inspiring watching babies learn is. Total nonsense, worst manipulation learning algorithm I've ever seen.
2 months of basically constant supervision and no applications to real-world tasks at all. It's like grad school all over again
21
1
159
4
2
44
🏆Nominated for Best Manipulation Paper at
@ieee_ras_icra
#HopMan
outlines a new paradigm towards a generalizable universal agent ➡️No RL, no imitation, only zero-shot translation of Human Interaction Plans😯
⏲️Today
- Talk: WeAA1-CC.2 <10:30-12:00>
- Poster: <13:30-15:00>
Excited to share our latest on generalizable zero-shot manipulation in the real world!
We can train a single goal-conditioned policy that scales to over 100 diverse tasks in unseen scenarios, including real kitchens and offices.
w/
@Vikashplus
Abhinav Gupta
@shubhtuls
1/n
2
28
152
3
5
43
📢 #𝗠𝘆𝗼𝗦𝘂𝗶𝘁𝗲 𝟮.𝟬
Towards generalizable agents ✍️🤾♀️🤳🏃♀️
If generalization was easy
it would be stupid for evolution
to discover its solution in
complex neural & morphological architectures!
#MyoSuite
2.0 is packed with evolutionary secrets!💰✨
(1/N)🧵
1
9
40
Three projects at
#ICML2023
this week w amazing students & colleagues🙏
1⃣ 𝗠𝘆𝗼𝗗𝗲𝘅➡️
@CaggianoVitt
@SudeepDasari
2⃣ 𝗟𝗜𝗩➡️
@JasonMa2020
@yayitsamyzhang
@obastani
@dineshja
3⃣ 𝗩𝗶𝘀𝘂𝗮𝗹 𝗗𝗲𝘅𝘁𝗲𝗿𝗶𝘁𝘆➡️
@taochenshh
M.Tippur, S.Wu, E.Adelson
@pulkitology
Details🧵👇
1
3
40
#ImitationLearning
is progressing fast. Will it overshadow
#Sim2Real
's success? ⛈️
Let's debate 🗣️. We are creating a safe platform at
@ieee_ras_icra
w\
@MyoSuite
.
📩Submissions open
1
9
38
Embodied beings🏃discussing embodied intelligence🦾🤖
@ieee_ras_icra
Topics:
- 3 fingers on an 🐘elephant?🤯
-
@Vikashplus
on an 🐘 in 2013🤫
- Humanoid: future & timelines
- Legs, Hands: mistake/opportunity?
@pulkitology
@mukadammh
@shaneguML
@CaggianoVitt
@yuvaltassa
@erikfrey
1
4
39
Officially released!
#ROBEL
's a bet to stimulate & facilitate real-world progress in robotics.
It has benefitted a wide diversity of projects (from dexterous manipulation to agile locomotion) & breath of researchers (from novice to theoreticians) at
@UCBerkeley
and
@GoogleAI
.
Check out ROBEL, an open-source platform for cost-effective robots and curated benchmarks designed to facilitate research and development on real-world hardware. Learn more below ↓
2
113
266
2
13
37
⚡️ALERT: Postdoc opportunities available at
@MetaAI
📢 We are especially looking for robot learning candidates
🤖🤖🤖🤖🤖🤖🤖🤖🤖🤖🤖🤖🤖🤖🤖🤖🤖🤖🤖
Send me a ping if you curious and wanna know more ...
0
6
37
Can't wait to finally release what I've been working on over the last 2 years.
Robot loving people -- Stay tuned. Something exciting is gonna happen 😁
1
0
37
Congratulations
#CoRL2022
best papers winners!
@hausman_k
,
@pathak2206
,
@dineshjayaraman
and collaborators
0
3
36
Here I though they we supposed to help clean instead !
(-ve result from an overnight run)
1
0
35
🔍In search of something interesting? Stay tuned
A bunch of interesting releases lined up on
@MyoSuite
!
1
2
34
Last week, we presented our *single-task* multi-scene agent
Today we are announcing
#CACTI
- a scalable *multi-task* multi-scene framework that delivers agents capable of visuomotor skill generalization to 100s of scenes ➡️
1/N🧵
1
7
32
Robotics is a journey that demands tools at the intersection of hardware and software. TRI betting into restricting mobility for better manipulability. An intriguing design choice... What benefits does this have over wheeled locomotion?
5
2
32
Very excited to be featured in Robotics Talent Spotlight. People will be tuning-in from around the world to discuss our present and future plans in Embodied AI.
And, we are hiring for both research and engineering positions!
1
2
33
Reproducibility in robotics research is necessary but has proven challenging, especially with hardware results.
we took a lot of effort into packaging our research on
#ALOHA
for reproducibility. And its is spreading! First signs coming in. A few more are in works...
2
5
32
Hidden behind the limelight of the new electric ATLAS,
@BostonDynamics
also unveiled impressive dynamic manipulation abilities.
Given the progress in high-level semantic reasoning via VLMs, task-specific robust low-level (MPC) planners are the key missing ingredient!
BD? 👀
6
4
32
#RoboAgent
is -
⚡️ 6x better than
🤯 needs 18x less data!
Appearing today in the Robot-Learning workshop
@NeurIPSConf
(Hall-B)
w
@mangahomanga
@m0hitsharma
@jdvakil
Abhinav Gupta,
@shubhtuls
#𝗥𝗼𝗯𝗼𝗔𝗴𝗲𝗻𝘁 -- A universal multi-task agent on a data-budget
💪 with 12 non-trivial skills
💪 can generalize them across 38 tasks
💪& 100s of novel scenarios!
🌐
w/
@mangahomanga
@jdvakil
@m0hitsharma
, Abhinav Gupta,
@shubhtuls
4
67
248
0
4
31
A perfect example why - Robotics is a quest at the intersection of hardware and software.
LocoMan
= Quadrupedal Robot + 2 * Loco-Manipulator
Powered by dual lightweight 3-DoF Loco-Manipulators and the Whole-Body Controller, LocoMan achieves various challenging tasks, such as manipulation in narrow spaces and bimanual-manipulation.
👇👇👇
5
46
200
0
4
31
Real world is the litmus test for Robotics.
One can't assume information or cheat. Let's bring robots out of the labs and simulations into the real world.
Shout out to
@chris_j_paxton
and team who have been passionately working hard over an year to make it happen!
The future of robot butlers starts with mobile manipulation.
We’re announcing the NeurIPS 2023 Open-Vocabulary Mobile Manipulation Challenge!
- Full robot stack ✅
- Parallel sim and real evaluation ✅
- No robot required ✅👀
4
73
321
2
7
31
Visiting beautiful Boston for a few days for a few talks at
@BostonDynamics
@MITCSAIL
@amazon
robotics,
@Northeastern
.
🤓
0
0
29
"Touch at the very least refines, & at the very best disambiguate visual estimates during in-hand manipulation"--
@Suddhus
A pearl of crucial wisdom, yet challenging to convincingly leverage & exploit in robotic dexterous manipulation.
Reliable surface sensing is the key!
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation
paper page:
Neural perception with vision and touch yields robust tracking and reconstruction of novel objects for in-hand manipulation.
3
37
175
1
2
29
An independent study identifies
#MuJoCo
’s as best and easy to use simulator for deformable objects studies 💪
We recorded data of two robots performing dynamic (left) and quasi-static (right) cloth manipulation. Why?
The dynamic motion produces a large deformation on cloths, really challenging to simulate!
The quasi-static requires accurate simulation of frictional and inertia forces.
1
1
11
0
3
29
✨Curiosity around the sample efficiency of #𝗥𝗼𝗯𝗼𝗔𝗴𝗲𝗻𝘁 triggered multiple requests on its (i.e.
#RoboSet
) training data distribution🤔
Attached is a stop motion of the initial object configurations for a particular scene
#𝗥𝗼𝗯𝗼𝗔𝗴𝗲𝗻𝘁 -- A universal multi-task agent on a data-budget
💪 with 12 non-trivial skills
💪 can generalize them across 38 tasks
💪& 100s of novel scenarios!
🌐
w/
@mangahomanga
@jdvakil
@m0hitsharma
, Abhinav Gupta,
@shubhtuls
4
67
248
0
1
29
An approachable blog of our
#ALOHA
and
#ACT
efforts -
A few follow up works from us that also might be of interest
- MTACT -
- MobileAloha -
✍️ Are you curious about Deep Learning and robotics?
The Action Chunking Transformer by
@tonyzzhao
et al. is a fascinating and critically important piece of research!
As a weekend fun project, I wrote a blog post to help others and me understand it:
2
3
34
0
4
29
🤯 How did we train a universal agent with merely 7k trajectories?
Join
#RoboAgent
team today
@ieee_ras_icra
- Big Data in Robotics and Automation Session at 4.30 pm (CC-313)
- Poster Session 10.30 am - 12 pm (Board 03.05)
#𝗥𝗼𝗯𝗼𝗔𝗴𝗲𝗻𝘁 -- A universal multi-task agent on a data-budget
💪 with 12 non-trivial skills
💪 can generalize them across 38 tasks
💪& 100s of novel scenarios!
🌐
w/
@mangahomanga
@jdvakil
@m0hitsharma
, Abhinav Gupta,
@shubhtuls
4
67
248
1
5
28
🪃 Heading to
#ICRA2023
this week6⃣contributions from our group+collaborators!
Feeling extremely grateful and lucky to be a part of these collaborations
A thread with details🧵👇
2
3
28
🌟Exciting updates on recent submissions🌟
🙏Grateful for the students, mentors, & collaborators for these fun projects -
@CaggianoVitt
@SudeepDasari
@JasonMa2020
@yayitsamyzhang
@obastani
@dineshjayaraman
@ZoeyC17
cameron
@tonyzzhao
@abhishekunique7
@chelseabfinn
@svlevine
3
2
28
Time and again
@UnitreeRobotics
has changed the game with Robotics’s capabilities at a reasonable cost.
For H1
- form factor is lean
- motors are strong
- battrey reasonable
- cost needs to come down a bit
Reliability for continuous deployment - jury’s out there testing ⏳
Unitree H1 Breaking humanoid robot speed world record [full-size humanoid] Evolution V3.0 🥰
The humanoid robot driven by the robot AI world model unlocks many new skills!
Strong power is waiting for you to develop!
#Unitree
#AI
#subject3
#BlackTech
40
226
912
1
2
28
𝙈𝙤𝙩𝙤𝙧 𝙗𝙚𝙝𝙖𝙫𝙞𝙤𝙧 𝙞𝙨 𝙖 𝙧𝙚𝙖𝙙 𝙤𝙪𝙩 𝙤𝙛 𝙞𝙣𝙩𝙚𝙡𝙡𝙞𝙜𝙚𝙣𝙘𝙚
Over the last 2 years, I have been focusing on understanding physiological behavior synthesis, & building
@MyoSuite
.
In this talk, I summed up the lessons learned and its relevance to Robot Learning
The first workshop on Learning Dexterous Manipulation at
@RoboticsSciSys
is starting now! Check out our speaker lineup at or tune in via zoom at if you are not in person.
0
6
20
3
5
28
Another big realization of 2023
🤏compliant gripper prongs are the easiest/simplest change that will give you the biggest performance boost.
#ROI
Now try communicating & getting a paper accepted about that !!
Dyson Robot Learning Lab is Hiring full-timers and interns! 🤖
1x Research Scientist:
1x Data Engineer (Data Collection & ML Training):
3x PhD Internship:
Come join our lab of 12; located in London, UK 🇬🇧
8
60
325
2
3
27
Daily Reminder:
- These aren't renderings.
- These are real.
It's happening ▶️
2
2
29
Team
#RoboAgent
is honored to be featured as a spotlight story on
@CarnegieMellon
's homepage 🙏
🌐
w
@mangahomanga
@m0hitsharma
@jdvakil
Abhinav Gupta,
@shubhtuls
#𝗥𝗼𝗯𝗼𝗔𝗴𝗲𝗻𝘁 -- A universal multi-task agent on a data-budget
💪 with 12 non-trivial skills
💪 can generalize them across 38 tasks
💪& 100s of novel scenarios!
🌐
w/
@mangahomanga
@jdvakil
@m0hitsharma
, Abhinav Gupta,
@shubhtuls
4
67
248
0
6
27
The next version of our
#ALOHA
➡️
#MOBILE_ALOHA
@tonyzzhao
@zipengfu
& co. outdid themselves! Congratulations.
💗My favorites: Bloopers toward the ends of the video. Don't miss out
Introducing 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀🏄 -- Hardware!
A low-cost, open-source, mobile manipulator.
One of the most high-effort projects in my past 5yrs! Not possible without co-lead
@zipengfu
and
@chelseabfinn
.
At the end, what's better than cooking yourself a meal with the 🤖🧑🍳
234
1K
5K
0
2
26
- 0.6 m/s
- model based
- motion plan + state machine
- addition of arm sways etc for naturalness
=> immersive nibble locomotion. Would be exciting to see some error recover
@julianibarz
3
2
26
Quite surprised by reception of
#RoboAgent
. Most knew about it in details, had great feedback, and share new extensions they are working on.
Kudos to
@tonyzzhao
,
@mangahomanga
@m0hitsharma
@jdvakil
@ZoeyC17
@ZhaoMandi
student leads on this research stream
#ACT
#GenAug
,
#CACATI
🤯 How did we train a universal agent with merely 7k trajectories?
Join
#RoboAgent
team today
@ieee_ras_icra
- Big Data in Robotics and Automation Session at 4.30 pm (CC-313)
- Poster Session 10.30 am - 12 pm (Board 03.05)
1
5
28
0
3
27
If you can induce a training distribution that is super set of your test distribution, it’s almost stupid to do anything but
#sim2real
.
We are organizing an
@ieee_ras_icra
workshop on Expanding frontiers of
#sim2real
-
Details to follow soon
We trained ANYmal to go into confined spaces such as under collapsed buildings. To be presented at
#ICRA2024
Title: Learning to walk in confined spaces using 3D representation
Arxiv:
Video:
Summary Page:
10
67
353
0
2
27
To further mark the role of off-domain datasets in robotics -
Ecstatic to announce📢next in the line of our
𝗙𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻 𝗠𝗼𝗱𝗲𝗹𝘀 𝗳𝗼𝗿 𝗥𝗼𝗯𝗼𝘁𝗶𝗰𝘀
from our group & collaborators
#RRL
➡️
#PVR
➡️
#R3M
➡️
#VIP
➡️
#H2R
▶️#𝗟𝗜𝗩
🧵👇
Excited to share our
#ICML2023
paper ✨LIV✨!
Extending VIP, LIV is at once a pre-training, fine-tuning, and (zero-shot!) multi-modal reward method for (real-world!) language-conditioned robotic control.
Project:
Code & Model:
🧵:
2
44
163
1
4
26
@svlevine
Behind every successful real-world training is a reset mechanism that no one talks about. Highlighting the unsung backstage heroes of our show.
0
2
25
An unpopular opinion in hot humanoid era 🦾🦿
My bet is on
@hellorobotinc
as a safe and welcoming form factor most poised to enter unstructured home environments. It’s a step in the right direction but more work is needed.
We still have a lot of work in front of us but our dream of home robots that benefit everyone is coming into focus!
0
10
30
3
3
24
RL suffers from ineffective search due to credit assignment problems in high dimension spaces.
Not reward tuning, but exploration based methods like
#LATTICE
,
#DepRL
(
@rlfromlux
), SAR (Cameron), MyoDex (
@CaggianoVitt
@SudeepDasari
) are breaking new grounds. Exciting times!!!
Tomorrow Wed 13 Dec 10:45 a.m. CST,
@a_marinvargas
and I will present Lattice at
#NeurIPS2023
Lattice is a new latent exploration method for Reinforcement Learning! Also top-ranked solution of the
@MyoChallenge
!
Poster
#1401
:
1
8
35
0
4
25
We are 🏃🏃♂️🏃♀️running 🏃🏃♂️🏃♀️ live with the most requested feature - 𝗠𝘆𝗼𝗟𝗲𝗴𝘀
🔥MyoSuite 1.5 released🔥
Introducing 𝗠𝘆𝗼𝗟𝗲𝗴𝘀 (most requested feature)
🦵80 muscles, 20 degrees of freedom
💢Accurate muscle+contact+inertial dynamics
✅Locomotion baselines coming soon
👉1click-webdemo
2
12
49
1
4
24
I really like the transparency in communication
- It's autonomous.
- It's 16% human speed!
- It's wired (no video gimmick to hide it)
=> It's amazing 👍
Figure 01 is now completing real world tasks
Everything is autonomous:
-Autonomous navigation & force-based manipulation
-Learned vision model for bin detection & prioritization
-Reactive bin manipulation (robust to pose variation)
-Generalizable to other pick/place tasks
64
186
985
3
2
24
𝗗𝗮𝗿𝘄𝗶𝗻'𝘀 𝘁𝗵𝗲𝗼𝗿𝘆 𝘄𝗼𝘂𝗹𝗱 𝗯𝗲 𝘄𝗿𝗼𝗻𝗴 if natural selection picked such complex embodiment to just make computation a challenge for our🧠
Morphological complexity is what enables our nimbleness
𝗔𝗜 𝗮𝗴𝗲𝗻𝘁𝘀 𝗻𝗲𝗲𝗱 𝗺𝗼𝗿𝗽𝗵𝗼𝗹𝗼𝗴𝗶𝗰𝗮𝗹 𝗽𝗿𝗶𝗼𝗿𝘀
Want natural motion with RL and muscles, but mocap data is limiting? biological objectives + constraints now achieve natural walking with 90 muscles!
Thanks
@tgeijten
for contributions with
#Hyfydy
and
@CaggianoVitt
&
@Vikashplus
for
#MyoSuite
preprint
11
61
279
0
4
24
Appearing in
#RSS2023
this week, 4⃣strong papers from our group!
From incredible lead authors 💪
🎯
#GenAug
-
@ZoeyC17
(best system paper nomination)
🎯
#ALOHA
-
@tonyzzhao
🎯
#SAR
- Cameron Berg
🎯
#LIV
-
@JasonMa2020
Details in thread 🧵🧵🧵
(all times GMT+9, Korea local time)
2
2
24
PhD students should try to organize at least 1 workshop before they graduate.
➡️ Best opportunity to mark+catalyze your research subfield
➡️ Best way to build academic network
➡️Best opportunity to closely interact with senior members
➡️Teaches skills school/publications doesn't
Interested in encouraging debate/discussion on a focussed topic in Robot Learning?
Consider submitting a workshop proposal to CoRL this year.
📅 Deadline: June 22nd, 2023
⚡️Decisions: June 30th, 2023
🎤Workshops: November 6, 2023
0
1
15
0
3
24
📢
#VIP
: Pre-trained reusable rewards & visual representation for general robot learning!
VIP is pre-trained purely on human videos & effectively generalizes to unseen robot tasks bringing new life to paradigms seeking visual rewards - TrajOpt, OnlineRL, OfflineRL
Details ⤵️
Excited to share VIP, a self-supervised visual reward and representation pre-trained on diverse human videos!
VIP’s frozen reward and rep. can solve diverse unseen robot tasks using TrajOpt, online RL, and enables real-world few-shot offline RL!
🧵:
2
40
169
2
2
23
We'r organizing a workshop at the intersection of NeuroMechanics, Robotics, & Deep Learning:
📢#𝗠𝘆𝗼𝗦𝘂𝗶𝘁𝗲 𝟮.𝟬 🔥
💪amazing speaker lineup!
🔎May 29th | South Gallery Room 29 | 9:00-13:00
w/
@CaggianoVitt
@MassimoSartori8
@GDurandau
@SeungmoonS
We're gearing up for
#ICRA2023
in London next week with an exciting speaker line-up🗣️
👉E. Todorov
👉
@pulkitology
👉
@nidhi_s91
👉 A. Seth
👉 F. Valero-Cuevas
👉
@Vikashplus
Join us at @ Neuromechanics Meets Deep Learning workshop
1
8
33
1
5
23
✨2023 highlight from our group✨
#RoboAgent
-- an efficient multi-task, multi-skill universal robotic agent
A culmination of over 2 years of collaborative research, & engineering
@AIatMeta
&
@CMU_Robotics
pushing forward the frontier of robotics
As we wrap up what has been a monumental year for AI, what was the most exciting advancement, publication, research breakthrough or AI-powered product you saw in 2023?
64
30
229
1
1
23
A NYT piece covering our recent push on dexterous manipulation with low cost commodity modular hardware (
@svlevine
@shaneguML
,
@abhishekunique7
,
@aravindr93
), among other efforts at *Robotics at Google* (
@sandraorion
,
@xenotaur
@andyzengtweets
)
0
4
23
Game on
@Figure_robot
@Tesla_Optimus
@agilityrobotics
@TheSanctuaryAI
@1x_tech
This year is going to be fun!
All VCs go burrr 💸🦅
0
0
22
Have you leveraged & benefitted from
@pytorch
?
we are presenting our agent's framework:
@torchrl1
- a one-stop solution for everything from datasets to agents & beyond
Meet the team at
@iclr_conf
this week (schedule below)
2
4
21
In the pursuit of dexterity
❌ most are betting on sim2real on single-tasks
💪 our 𝗱𝗲𝘅𝘁𝗲𝗿𝗼𝘂𝘀 𝗗'𝗛𝗮𝗻𝗱 is 𝗰𝗼𝗻𝘁𝗶𝗻𝘂𝗮𝗹𝗹𝘆 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴 in the real world to generalize to new tasks & objects
In
@corl_conf
today Nov 9, 2:45-3:30 pm
Can we get dexterous hands to learn efficiently from images entirely in the real world? With a combo of learned rewards, sample-efficient RL, and initialization from data of other tasks, robots can learn skills autonomously in a matter of hours:
A 🧵👇
1
29
178
1
3
22
A big move towards the pursuit of uncovering fundamentals behind 𝒉𝒖𝒎𝒂𝒏 𝒎𝒐𝒕𝒐𝒓 𝒊𝒏𝒕𝒆𝒍𝒍𝒊𝒈𝒆𝒏𝒄𝒆!
(or 𝒉𝒖𝒎𝒂𝒏 𝒂𝒕𝒉𝒍𝒆𝒕𝒊𝒄 𝒊𝒏𝒕𝒆𝒍𝒍𝒊𝒈𝒆𝒏𝒄𝒆 in the words of Marc Raibert from
@BostonDynamics
)
🚀#𝗚𝗼𝗼𝗴𝗹𝗲𝗗𝗲𝗲𝗽𝗠𝗶𝗻𝗱 joins #𝗠𝘆𝗼𝗦𝘂𝗶𝘁𝗲🚀
@MyoSuite
is ecstatic to announce support from
@GoogleDeepMind
toward achieving human-level dexterity & agility in physiological embodied agents via
#MyoChallenge
🎯Proud of your algo? 🏹10 days left to prove it
1
12
55
0
2
22
Today at
#ICLR2023
, we are presenting two works ⬇️
🔥
#VIP
(top 25% paper):
A self-supervised visual reward + representation pre-trained on diverse human videos
1⃣Oral 1 Track 5: Reinforcement Learning, 10:30am
2⃣Poster: Mon 11:30AM (MH1-2-3-4
#118
)
VIP enables a simple and practical real-world few-shot offline RL pipeline: just do reward-weighted regression (RWR) with VIP’s reward and the representation! With VIP, offline RL is as simple as BC but far more effective.
1
1
3
1
1
21
Nonconvex shapes require expressive
𝙛𝙞𝙣𝙜𝙚𝙧 𝙜𝙖𝙞𝙩𝙞𝙣𝙜 for in-hand reorientations
D'Hand can now *autonomously* learn such behaviors in a few hours
🙀Trained on-hardware
👁���Visual inputs
🎯Learned rewards
🫰 Data frugality & reuse💪
🧵👇
2
4
21
In the 𝗟𝗟𝗠 wave, we are overlooking the potential of 𝗚𝗲𝗻𝗔𝗜 in Robotics. Why?
🧠Not playing (TAMP), moving the chess 🤏pieces (Moravec's paradox) is the challenge.
💸 LLMs assume, but GenAI aids stable low-level behaviors by baking real-world invariances in data for free!
0
3
19
𝗛𝟮𝗥 will be featuring twice‼️ @
#ICRA223
today
🥇 (𝘀𝗽𝗼𝘁𝗹𝗶𝗴𝗵𝘁) Pretraining for robotics | 14:00
🥈(𝘁𝗼𝗽 𝟭𝟱%) Effective Representations & prior for robot learning | 15:30
w\
@mangahomanga
Abhinav Gupta
@shubhtuls
Human possess a remarkable ability to perform a broad spectrum of tasks with varying degrees of precision. Can we imparting similar levels of versatility to robots merely by watching human videos?
📢 H2R - an agent capable of common coarse-manipulation skills
1/N🧵
2
12
68
0
3
19
Most good research 𝗱𝗶𝗲𝘀 at the quarter mile mark!
Pursue a Research Direction🎯
Not Research Projects✖️
Credit: Abhinav Gupta,
@pathak2206
at
@CMU_Robotics
Retreat.
0
1
20
**Human are the rate limiting step in scaling robot learning**
-- a broader article placing our latest work on uninterrupted multi-task robot learning, MTRF (), in the larger context of real world scalable robot learning.
I wrote a short, non-technical article summarizing some recent progress in robotic learning, particularly how robots can learn (1) from large offline datasets; (2) rapidly adapt online in new settings; (3) learn autonomously:
Robots like the one below:
2
92
420
1
6
20
We're gearing up for
#ICRA2023
in London next week with an exciting speaker line-up🗣️
👉E. Todorov
👉
@pulkitology
👉
@nidhi_s91
👉 A. Seth
👉 F. Valero-Cuevas
👉
@Vikashplus
Join us at @ Neuromechanics Meets Deep Learning workshop
1
8
33
0
0
20