Jiafei Duan
@DJiafei
Followers
1,382
Following
805
Media
140
Statuses
517
Robotics and AI PhD Student @uwcse @uw_robotics | @NVIDIA Intern |AI Research @ASTARsg | BEng from @ntueee . Research in robot learning and embodied AI
Seattle, WA
Joined January 2021
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
England
• 372246 Tweets
Noah Lyles
• 249973 Tweets
Rotherham
• 235602 Tweets
ヒロアカ
• 157430 Tweets
Polônia
• 123185 Tweets
NBA Twitter
• 103761 Tweets
World Champion
• 81072 Tweets
Corinthians
• 69362 Tweets
フワちゃん
• 68023 Tweets
Gabi
• 61265 Tweets
堀越先生
• 61235 Tweets
Thompson
• 55168 Tweets
Carol
• 48376 Tweets
Rosamaria
• 46469 Tweets
Bolt
• 41061 Tweets
Evandro
• 39504 Tweets
Wesley
• 39191 Tweets
Peillat
• 38044 Tweets
Jamaica
• 36295 Tweets
GABRIELA
• 30952 Tweets
Seville
• 29637 Tweets
#VoleiNoSporTV
• 29206 Tweets
38x36
• 28985 Tweets
thaisa
• 26799 Tweets
Tamworth
• 23270 Tweets
Simbine
• 22114 Tweets
Kerley
• 18778 Tweets
Greggs
• 17237 Tweets
Juventude
• 15459 Tweets
Solari
• 14409 Tweets
#めざましテレビ
• 14022 Tweets
Peñarol
• 13377 Tweets
神さま学校の落ちこぼれ
• 12260 Tweets
ケイくん
• 11264 Tweets
Pinned Tweet
🚀 Excited to share our latest work: MANIPULATE-ANYTHING! 🦾 This scalable method pushes the boundaries of real-world robotic manipulation through zero-shot task execution and automated BC data generation. Here's a quick overview:👇
6
59
277
For large-scale robotic deployment🤖 in the real-world 🌏, robots must adapt to changes in environment and objects. Ever questioned the generalizability of your robot's manipulation policy? Put it to the test with The Colosseum 🏛️. Check out our project:
6
41
244
Humans use pointing to communicate plans intuitively. Compared to language, pointing gives more precise guidance to robot behaviors.
Can we teach a robot how to point like humans? Introducing RoboPoint 🤖👉, an open-source VLM instruction-tuned to point.
Check out our new work:
6
34
155
Grad school application season is here.
I always find that PhD application is about outsmarting the smart ones, so I wrote this article to share some of my experiences in PhD application.
Best of luck!
4
20
136
After receiving multipe offers from top computer science/AI programmes globally, I have decided to pursue my PhD study at the
@UW
,
@uwcse
.
I will be joining the legendary roboticist, Professor Dieter Fox's group at University of Washington!
#phdlife
#AcademicChatter
9
2
122
First look at
@UnitreeRobotics
G1 humanoid at ICRA!
Small and compact, and really good balancing and stability.
Great platform for humanoid research
5
16
107
The upper bound of a VLA model is probably 90% of all tasks, but there is going to be inherently that 10% of robotics tasks which couldn’t be solved without the understanding of other multimodal inputs (such as audio, tactile and etc). 🤖
1
13
102
🎉Happy to share that I will be joining NVIDIA as a 2024 summer research scientist intern working on robotic manipulation. See you in Santa Clara and Seattle!
*Hope to meet Jensen in-person
7
1
97
Feels surreal, but here I am. Travelling halfway across the 🌎 to do my
#PhD
for the next 5 years.
A mix feeling of uncertainty and excitement for the unknown future.
#phdlife
@PhDchatter
1
3
70
Try out
#LumaDreamMachine
for robotics action generation, even though there are artifacts in the object generated, but I would say that the kinematics of the robot motion is pretty good. Can we use for robotics data?
4
6
52
If you attended Prof. Dieter Fox's talk on 'Where is RoboGPT?' At
@RoboticsSciSys
, you'll recall his sharing of our work, RoboPoint a domain-specific spatial VLM for generating intermediate representations beneficial for robotics tasks. We are pleased to announce that our live
Humans use pointing to communicate plans intuitively. Compared to language, pointing gives more precise guidance to robot behaviors.
Can we teach a robot how to point like humans? Introducing RoboPoint 🤖👉, an open-source VLM instruction-tuned to point.
Check out our new work:
6
34
155
0
9
37
Will be presenting this work at the
#ICRA2024
Future roadmap for manipulation skills workshop tomorrow! I will be the first to present, come check it out and talk to me about Benchmarking for Robotics Manipulation. 🦾
For large-scale robotic deployment🤖 in the real-world 🌏, robots must adapt to changes in environment and objects. Ever questioned the generalizability of your robot's manipulation policy? Put it to the test with The Colosseum 🏛️. Check out our project:
6
41
244
0
1
33
I will be starting my NVIDIA internship at Seattle Robotics Lab for the summer! Let’s see how much the stock will be after I end the internship 😂📈
0
1
33
#RSS
@RoboticsSciSys
has definitely become one of my favorite conferences! I met many old and new friends and saw many awesome works.
0
1
35
That’s a wrap for 2023!
Happy New Year everyone 🎊.
All the best to those that are still working for RSS in the lab!
1
0
32
Happy to share that our work got accepted at
#ICLR2024
as a spotlight paper(5%)!
📣Humans intuitively focus on things essential for the task at hand, inspired by this idea💡. We're excited to explore whether an Embodied AI agent can develop a more task-driven visual perception through such a 'filtering' mechanism. 🤖✨
🔍 Check out our latest work on the
0
2
21
1
1
27
Foundation models in robotics can probably handle 90% of real-world tasks given enough data. However, it's the remaining 10% that truly matters, and this gap is unbridgeable without other modalities. Hence, I am excited about our new
#RSS
work on tactile-based LVLM for robotics!
TLDR: Octopi is a LVLM that leverages both tactile representation learning and large vision-language models to predict and reason about vision-based tactile inputs with minimal language fine-tuning.
Project page:
Paper:
0
5
19
0
4
29
With so many exciting work that came out over the past few weeks, we will like to do a soft release of our benchmark🧑💻
Everything you would need to get started with Colosseum are well documented here:
But feel free to contact us for more questions!
For large-scale robotic deployment🤖 in the real-world 🌏, robots must adapt to changes in environment and objects. Ever questioned the generalizability of your robot's manipulation policy? Put it to the test with The Colosseum 🏛️. Check out our project:
6
41
244
1
9
28
Had a blast presenting our work at
@corl_conf
2023 and making new friends in robot learning. Huge shoutout to the organizers for a fantastic conference!
#CoRL2023
0
0
24
📢Heading to
@corl_conf
in Atlanta to present "AR2-D2: Training a Robot without a Robot." Learn about how to gather training data for robotic demonstrations without needing a physical robot?! 🤖 Join Poster Session 2: RL/IL, Nov 7, 4:15-5:00 PM for a live demo!
#Robots
#AR
#data
1
3
20
📣Humans intuitively focus on things essential for the task at hand, inspired by this idea💡. We're excited to explore whether an Embodied AI agent can develop a more task-driven visual perception through such a 'filtering' mechanism. 🤖✨
🔍 Check out our latest work on the
Embodied-AI 🤖 models employ general-purpose vision backbones such as CLIP to encode the observation. How can we have a more task-driven visual perception for embodied-AI?
We introduce a parameter-efficient approach that selectively filters visual representations for Embodied-AI
1
27
125
0
2
21
"A Survey on Machine Learning Approaches for Modelling Intuitive Physics" has been accepted to
@IJCAIconf
2022, survey track (18% acceptance rate).
Check out the paper and resources.
Paper:
Webpage:
#MachineLearning
#Physics
1
2
19
PIP has been accepted to
#ECCV2022
!
This work helps to improve physical reasoning through mimicking the selective temporal attention of humans during mental simulations.
Paper:
Website:
@SamsonYuBaiJian
@bihan_wen
@soujanyaporia
1
3
17
I am more and more convinced by the phrase, this year is going to be the year of robotics 🤖
Let’s think about humanoid robots outside carrying the box. How about having the humanoid come out the door, interact with humans, and even dance?
Introducing Expressive Whole-Body Control for Humanoid Robots:
See how our robot performs rich, diverse,
94
209
1K
1
4
18
The year of robotics, begins with the tele-op systems for scaling up robot demonstration collection!
We tackle robotic manipulation with real-world imitation learning or sim2real approaches. But what if we merge the two?
Introducing CyberDemo: a novel pipeline that leverages simulated human demos to master real-world dexterous manipulation tasks.
5
40
199
2
3
18
Foundational models for robotics shouldn’t be just coming from one kind of data, this work is really cool! Combining the different modalities of data as different tasks might need different kinds of data to aid with learning.
How to deal with the vastly heterogeneous datasets and tasks in robotics? Robotic data are generated from different domains such as simulation, real robots, and human videos. We study policy composition (PoCo) using diffusion models.
project website:
7
33
154
0
4
17
We will be presenting this today at 1.30pm and poster at Senaatszaal!
Come talk to us!
@RoboticsSciSys
The upper bound of a VLA model is probably 90% of all tasks, but there is going to be inherently that 10% of robotics tasks which couldn’t be solved without the understanding of other multimodal inputs (such as audio, tactile and etc). 🤖
1
13
102
0
1
16
Check out our work that just won the Best Paper at the Ubiquitous Robotics Conference 2023.
Happy to share that our work Good Time to Ask was awarded the best paper award at Ubiquitous Robotics 2023! Thank you to Korea Robotics Society (KROS) for organizing and hope to keep in touch with everyone I met.
4
1
28
0
1
14
With the rise of low-cost robotics hardware these days, it reminded me of this robotic ‘arm’ I built in high school 😂, almost a decade ago. Still excited about it today!
1
0
16
This is definition of household robot that many of us use in the motivation section. I can see some light 💡at the end of the tunnel.
Introducing 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀🏄 -- Hardware!
A low-cost, open-source, mobile manipulator.
One of the most high-effort projects in my past 5yrs! Not possible without co-lead
@zipengfu
and
@chelseabfinn
.
At the end, what's better than cooking yourself a meal with the 🤖🧑🍳
236
1K
5K
0
1
14
Great work by
@mohito1905
!
GENIMA has shown an incredible level of generalizability in our Colosseum benchmark!
The formulation of action-generation as an image-generation is definitely a clever look at things, making this approach comparable to 3D-based next point prediction
Image-generation diffusion models can draw arbitrary visual-patterns. What if we finetune Stable Diffusion to 🖌️ draw joint actions 🦾 on RGB observations?
Introducing 𝗚𝗘𝗡𝗜𝗠𝗔
paper, videos, code, ckpts:
🧵Thread⬇️
2
32
152
0
1
15
Even though these tasks were all tele-operated, but I see the upper bound of household robotics and its potential! Really kudos to
@zipengfu
and
@tonyzzhao
on this!
Mobile ALOHA's hardware is very capable. We brought it home yesterday and tried more tasks! It can:
- do laundry👔👖
- self-charge⚡️
- use a vacuum
- water plants🌳
- load and unload a dishwasher
- use a coffee machine☕️
- obtain drinks from the fridge and open a beer🍺
- open
409
2K
7K
0
0
15
Kind of a late post. This quarter I am TAing CSE 599 AI vs IA for
@RanjayKrishna
. This course will explore the opportunities and intersections between HCI and AI research! Super interesting and insightful course, highly recommending everyone to take it in the next winter quarter.
1
0
14
With this newly accepted EMNLP paper, I guess I have went from CV to Cogsci to Robotics to NLP 😂
Truly multi-disciplinary
2
0
12
Happy to share that we have 0/2
#NeurIPS2022
papers accepted. Maybe it is like all the other great science discoveries that are just ahead of its time.
We will continue to work hard on it and release the code after the world has caught up with us. 😂
#phdchat
#AcademicChatter
0
1
11
Happy to share that our survey paper "A Survey on Machine Learning Approaches for Modelling Intuitive Physics" has been accepted to
@IJCAIconf
2022 survey track (18% acceptance rate!) w/
@ArijitNoobstar
(paper will be out soon)
#cogsc
#Physics
#AI
#AcademicChatter
1
4
12
Flying to
@RoboticsSciSys
, looking forward to meet up with new and old friends.
Anyone keen to meet up to chat about anything 🔥 in AI and Robotics, ping me up!
0
1
12
Mobile manipulation is definitely the future, this is really cool to see how navigate and manipulation works in harmony!
How to tackle complex household tasks like opening doors, and cleaning tables in real?
Introducing HarmonicMM: Our latest model seamlessly combines navigation and manipulation, enabling robots to tackle household tasks using only RGB visual observation and robot proprioception.
2
18
134
0
0
11
Thanks to
@TheAITalksOrg
for the invite! I will be sharing some of our recent work that focuses on how we could scale up ‘data’ the right way for building foundation models in robotics 🤖.
Feel free to join virtually!
0
1
9
This is the kind of things that gets both roboticist and everyone else excited about robotics and AI!
Boston Dynamics reveals its new, improved all-electric Atlas robot, and
@bheater
has all the details right here:
151
530
2K
0
0
10
Check out our AR2-D2 work being featured on MIT tech review alongside many important works on collecting robot data in this era of robot data crunch!
0
0
10
I am always a huge fan of all the demos coming out from
@xiaolonw
group! Cool and also really practical!
Tesla Optimus can arrange batteries in their factories, ours can do skincare (on
@QinYuzhe
)!
We opensource Bunny-VisionPro, a teleoperation system for bimanual hand manipulation. The users can control the robot hands in real time using VisionPro, flexible like a bunny. 🐇
10
63
337
1
2
9
Check out our spotlight paper next week in ICLR! And code has been released for running.🏃
🎉 Very Excited to present our recent work on “Selective🔍 Visual Representations for Embodied-AI🤖” next week at ICLR in Vienna🇦🇹!!
📣📣Important update! Our code and pretrained models are now available through our project website 🌐: 🚀
👋Come to my
1
8
52
0
1
9
I had the privilege of attending an insightful keynote by the legendary
@chrmanning
. His guidance on how PhD candidates can excel in the era of Large Language Models (LLMs) was particularly enlightening. Additionally, it was an honor to present our research at
#EMNLP2023
. The
0
0
8
Grad school admission season are over. Got into my dream school, now is time to get back to work for
@NeurIPSConf
Season!
#Hustle
#AcademicTwitter
#phdchat
1
1
7
Huge congratulation to my advisor, Dieter for winning the John McCarthy Award! If you are at IJCAI, should definitely check out the talk!
#Invited
📢 Thrilled to announce Dieter Fox
@UW
invited talk🎙️on 'Toward Foundational Robot Manipulation Skills' at
#IJCAI2023
🗓️ 23 Aug, 9 AM 🤖 👉
#JohnMcCarthyAward
#IJCAIAwards
#Robotics
#MachineLearning
0
2
2
0
0
8
Mobile manipulation is hot🔥especially for bimanual 🦾🦾, really exciting to see more platforms supporting that!
Mobile ALOHA 🏄 is coming soon!
Special thanks to
@tonyzzhao
for throwing random objects into the scene, and
@chelseabfinn
for the heavy pot (> 3 lbs) !
Stay tuned!
9
59
377
0
0
8
After one full year of revision and rebuttal. Our survey paper on Embodied AI has finally been published at IEEE Transaction on Emerging Topics in Computational Intelligence.
#embodied
#AI
#AcademicChatter
2
1
7
Sometimes I feel that doing a PhD is like hiking. It is going to tough and painful as you are climbing up, but once you reach the peak and see the glorious view, it is all worthwhile!
@AcademicChatter
#phd
#hiking
1
0
8
Open AI is the best kind of AI. They even opensource the training scripts 🤯
OLMo is here! And it’s 100% open.
It’s a state-of-the-art LLM and we are releasing it with all pre-training data and code. Let’s get to work on understanding the science behind LLMs. Learn more about the framework and how to access it here:
29
350
1K
0
0
7
One of the reason, i choose uw might be due to this 😍
Alas, 3 weeks is far too short of a time to see such excellent and admirable people. I don't know half of you half as well as I should like, and I like less than half of you half as well as you deserve. 😉 1/2
1
21
227
0
0
7
Children are the best learning agent! Hence, we proposed "ABCDE: An Agent-Based Cognitive Development Environment." A work done by a final year student that I co-advised.
Happy to share that it has been accepted to the Embodied AI Workshop at this year's
#CVPR2022
!
Stay tune!
0
1
6
Will be at
#ICRA2023
I will be in London for
#ICRA2023
@ieee_ras_icra
. Excited to present my work on Good Time to Ask on 29 May at the Communicating Robot Learning across Human-Robot Interaction workshop!
0
3
33
0
0
7
After thinking about it, this will be my first-ever physical conference that I will be attending since I started doing research in undergraduate. 🤔
📢The registration for IJCAI-ECAI-22 is now open
#IJCAI2022
.
👉Meet who is who in artificial intelligence
#research
and
#innovation
.
👉Be part of the community in person.
👉Vienna, Austria 🇦🇹
W. Org:
@lucderaedt
@StefanSzeider
@magdalenaosimk
#AI
#ML
0
10
19
1
0
6
0
0
7
Grad school application season.
Things that I wished I would have known when I am applying for a CS
#PhD
.
#AcademicChatter
#gradschool
1
2
7
This is insane, research becomes a popularity contest.
Crazy AF. Paper studies
@_akhaliq
and
@arankomatsuzaki
paper tweets and finds those papers get 2-3x higher citation counts than control.
They are now influencers 😄 Whether you like it or not, the TikTokification of academia is here!
64
286
2K
0
0
7
14/🧵 This work could not be possible without my amazing collaborators and advisors,
@TonyWentaoYuan
@wpumacay7567
@YiruHelenWang
@ehsanik
@RanjayKrishna
and Dieter Fox. This project is supported by
@allen_ai
@nvidia
@uwcse
1
0
7
I think using iPhone or IPad📱to collect robot data might become a trend for democratizing robot learning at scale, just like our
Meet Dobb·E: a home robot system that needs just 5 minutes of human teaching to learn new tasks. Dobb·E has visited 10 homes, learned 100+ tasks, and we are just getting started!
Dobb·E is fully open-sourced (including hardware, models, and software):
🧵
13
111
497
0
1
7
Cool work! Looking forward to seeing this work opening new 'doors' in robotics!
Introducing Open-World Mobile Manipulation 🦾🌍
– A full-stack approach for operating articulated objects in open-ended unstructured environments:
Unlocking doors with lever handles/ round knobs/ spring-loaded hinges 🔓🚪
Opening cabinets, drawers, and refrigerators 🗄️
👇
30
104
776
0
0
6
The additional benefits of learning from 3D for visual imitation learning further being demonstrated here. Great work
@ZeYanjie
!
Introduce 3D Diffusion Policy (DP3), a simple visual imitation learning algorithm that achieves:
- 55.3% relative improvement on 72 simulated tasks, most with 10 demos
- 85% success rates on 4 real-world tasks, with 40 demos🥟🌯
Open-sourced!
Code/Data:
4
59
296
1
1
6
5/🧵 The action generation module produces a task-specific grasp pose or action code, leading to a temporary goal state. 🛠️
1
0
6