Kevin Zakka
@kevin_zakka
Followers
11,636
Following
868
Media
273
Statuses
2,183
phding @Berkeley_AI , interning @BostonDynamics
Berkeley, CA
Joined January 2016
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
#NuNew1stConcertDay2
• 137829 Tweets
South Africa
• 136960 Tweets
NUNEW DREAM CATCHER
• 114298 Tweets
ギラホス
• 40945 Tweets
マーリン
• 31863 Tweets
PHUWIN IN QINGDAO
• 31285 Tweets
小倉記念
• 31007 Tweets
ヴィーガン
• 26830 Tweets
LINGORM AROUND
• 26273 Tweets
関屋記念
• 25534 Tweets
DHRUV
• 24222 Tweets
大社高校
• 23009 Tweets
和田アキ子
• 22209 Tweets
#リエラと極上大冒険
• 22028 Tweets
ヘルナンデス
• 21437 Tweets
Ferhat Gedik
• 20426 Tweets
Sifan Hassan
• 19858 Tweets
契約解除
• 18685 Tweets
#iyikivarsınEren
• 18148 Tweets
Pinned Tweet
Introducing 𝗥𝗼𝗯𝗼𝗣𝗶𝗮𝗻𝗶𝘀𝘁 🎹🤖, a new benchmark for high-dimensional robot control! Solving it requires mastering the piano with two anthropomorphic hands.
This has been one year in the making, and I couldn’t be happier to release it today! Some highlights below:
55
236
1K
I do robotics + AI research and still I feel insane FOMO + left behind seeing all the advances in LLM world 😅
62
122
2K
Implemented a really neat ML algorithm called Neighbourhood Components Analysis in
@PyTorch
. It allows you to learn a linear transformation of your dataset that maximizes the performance of kNN. For high noise variance datasets, it handily beats PCA.
14
338
2K
Reading other people's code is so underrated. I learn so much from browsing repos in {Google / Deepmind / FAIR}'s Github.
21
72
910
JaX is def faster than PyTorch in every use case I've tried, but pretending like it's easier to use & write is a bit disingenuous. Just spent way more time than I expected writing a simple MLP class. Much simpler and you can print the model to debug it in PyTorch
26
65
832
The MuJoCo team just released a touch sensor using the new plugin API. It essentially aggregates all contact forces into an array of rectangular pixels to create an image.
Here's a braille demo from the amazing
@yuvaltassa
:
9
82
589
As promised, the new blog post is live: "kNN classification using Neighbourhood Components Analysis". Had a blast writing this one and making the visualizations.
5
113
562
Ben Katz's thesis is full of golden nuggets. In particular, I discovered today he had a really cool bilateral teleoperation system using two Mini Cheetah legs.
7
71
523
Excited to finally open-source 𝐦𝐢𝐧𝐤, a library for differential inverse kinematics in Python based on the MuJoCo physics engine.
7
76
532
Would be kind of cool to create a financial model where PhD students can write high-quality research code and maintain it (e.g. for new pytorch version) and get sponsored for it. GitHub already has the tooling in place 🙂
18
27
430
Life update:
I’m super excited to share that I’ll be joining
@Berkeley_EECS
as a PhD student next fall! Looking forward to tackling fun robotics problems with
@pabbeel
for the next few years 🤖 🧠
31
7
397
I tried doing IK for a shadow hand the first year of my PhD and utterly failed. Took me 5 minutes to set it up with my new diff IK library and it works out of the box at > 1 kHZ.
10
28
399
Copilot completing comments for me, suggests a username. I type the username into Github and get a legit profile of robotics phd student
11
39
366
MuJoCo 3.1.6 has just been released! MJX now comes with out-of-the-box support for height fields. Check out this joystick policy that
@the_real_btaba
trained (~2M steps per second on an A100).
Colab to reproduce:
4
51
365
MuJoCo 3.0 is out and it comes packed with amazing new features: TPU/GPU acceleration, non-convex geometries and deformable bodies!
Introducing MuJoCo 3.0: a major new release of our fast, powerful and open source tool for robotics research. 🤖
📈 GPU & TPU acceleration through
#JAX
🖼️ Better simulation of more diverse objects - like clothes, screws, gears and donuts
💡 Find out more:
17
303
1K
3
41
339
Just released a MuJoCo implementation of end-effector control with differential inverse kinematics. It's blazing fast, super readable, has 0 external dependencies and best of all, it's just a few lines of code:
4
35
324
I've updated the tutorial MuJoCo controller library with 3 examples on 3 different robots:
8
43
308
Still in progress but I've implemented Hyperband, a hyperparameter tuning algorithm, for
@PyTorch
. Currently supports fully-connected nets, conv nets coming soon. Should be useful for Kaggle competitions.
3
86
303
Just added a high-quality Rubik's cube model to the MuJoCo collection. You can play with it at:
12
35
276
Differential inverse kinematics controller on a Go1 quadruped regulating the pose of the root body and the position of the feet.
6
17
270
Just finished implementing collision avoidance in the diff IK solver as extra constraints in the QP. Really happy with the final result 😀
8
14
262
@geoffreyhinton
"If self-driving cars cut the roughly 40,000 annual U.S.
traffic fatalities in half, the car makers might get not
20,000 thank-you notes, but 20,000 lawsuits” - Russel et al 2015
3
26
251
Super stoked to release the very first episode of my Casual Robotics podcast: "Progress Towards General Purpose Robots" ft. the brilliant
@ericjang11
and
@peteflorence
.
11
27
252
It was a blast collaborating with the MuJoCo team on this project 🥳🤖! High quality models are going to be crucial for deploying sample-efficient algorithms in the real world. Looking forward to these robots being put to good use and for exciting community contributions!
11
21
235
Today was my last day as a student researcher at
@GoogleDeepMind
. It was a privilege to work again with
@andyzeng_
and
@peteflorence
on RoboPianist, exploring and pushing the state of the art in bimanual dexterity. With RoboPianist, I was introduced to the world of simulation and
5
2
231
>1 year of hard work and 0 citations to show for it. LM robotics paper all around me with hundreds of citations in a matter of days 🫠
20
2
227
Looks like I have something to keep me busy this summer 😁 A gem of a class: "Neural Net Training Dynamics" with high quality slides, notes and Colab notebooks using jax.
4
41
217
📣 I've generated MuJoCo models for all 1030 objects released in Google's Scanned Objects dataset and made them available at
The XML files are licensed under MIT, assets retain their original CC-BY 4.0 License.
6
18
213
🎁🤖Surprise drop: Unitree H1 humanoid is now available in MuJoCo Menagerie:
4
24
214
Increasingly feels like the only way to make non-trivial contributions in AI is to a) share and use the same battle-tested infrastructure and b) work with a diverse full-stack team where you can interchange ideas, datasets, models, and tricks of the trade..
Gato🐈a scalable generalist agent that uses a single transformer with exactly the same weights to play Atari, follow text instructions, caption images, chat with people, control a real robot arm, and more:
Paper: 1/
93
1K
5K
6
17
204
Center-of-mass regulation + feet/hands on the G1 humanoid.
Differential inverse kinematics controller on a Go1 quadruped regulating the pose of the root body and the position of the feet.
6
17
270
5
18
200
Y'all should have asked
@ylecun
before naming this :)
Meet Grand Teton, Meta’s next-generation platform for AI at scale! This year at OCP Summit 2022, we’re announcing Grand Teton, plus new innovations we’ve designed to support data centers as they advance to support new
#AI
technologies.
Get more details:
13
16
138
14
8
197
How can robots 🤖 learn from videos of humans, especially when humans perform the same task in different ways?
A 🧵 introducing our
#CoRL2021
paper "XIRL: Cross-embodiment Inverse RL".
Website & code:
1/
3
33
183
OpenAI shut down their robotics division to focus on more $ bets (generative modeling / GPT-N). Now those bets are directly driving or inspiring advances in robotics (socratic models, saycan) 😂
5
3
178
Safe bimanual control on ALOHA, powered by mink. Here I'm specifying collision avoidance between 1) left and right arm, 2) each arm and the 80/20 extrusions and 3) each arm and the table.
(yes laptop hiccuped a bit due to the sheer amount of collision computation, but I can
8
20
249
Finally have meshes loading properly in the MuJoCo browser visualizer 🥳
4
6
171
Boston Dynamics Spot and Franka Research 3 are now in Menagerie!
3
8
167
Pretty happy with the new gallery in the MuJoCo Menagerie README 😁
1
8
164
PhD Milestone: I'm officially a weak baseline in a paper 😅
How can we learn a reward for robotics tasks solely from diverse videos?
Introducing GraphIRL, a self-supervised method for learning a task reward w/ a graph abstraction from videos.
No manual reward design is required for goal-conditioned-RL!
(1/n)
2
38
174
4
3
158
Made a quick Colab notebook to explore OpenAI's CLIP and visualize which parts of an image activate given a certain caption.
2
21
159
Spent the break writing a research-friendly API for PyBullet, similar to `dm_robotics`. Nowhere near finished but wanted to share some images as I work through it in my downtime this year.
5
14
156
We have some pretty exciting models coming to Menagerie in the upcoming release! Stay tuned 😉
5
21
154
Starting "MuJoCo Weekly", a newsletter introducing MuJoCo users to new features, tips and tricks.
Follow along in this thread 👇:
4
16
146
2 iiwas avoiding each other. Directly inspired by a pending PR to pink ().
Just finished implementing collision avoidance in the diff IK solver as extra constraints in the QP. Really happy with the final result 😀
8
14
262
7
11
146
Transformers eating robotics
LLMs encode 1D tokens, vision transformers encode 2D image patches, but what's the right formulation for 6-DoF manipulation transformers?
Maybe 3D voxel patches?
🧵Checkout our
#CoRL22
paper – PerAct👇
Paper & Colab:
10
108
574
1
18
144
Just discovered you can finally insert latex equations in Google Docs 🤯
4
7
144
I’ve been working with MuJoCo + dm_control full-time for the last 2 months, and it’s been an absolute pleasure. Completely sold! So much respect for
@DeepMind
’s clarity of thought and foresight re:the API design. Will write up a blog post at a more appropriate time!
3
6
143
As the UR5 wrist approaches the cube, the collision avoidance constraint scales down the velocities in the IK solver to produce a nice damping effect. Without it, the robot rams into the cube producing very large forces.
1
6
144
3 weeks left of my masters and thesis almost complete. Excited for what’s next 🤖
5
4
142
So my PhD friend
@smithlaura1028
has been casually using a reset policy she trained in simulation with reinforcement learning 2 years ago 🤯!! I couldn't believe it was a learned policy when I first saw it. See for yourself 👇
3
13
134
One of my absolute favorite professors and roboticists cited me in his lecture, my day has been made 😳🤯!!
5
2
136
Woke up to an amazing email today — I’ve been named 2021 Centennial TA for teaching contributions to CS231n ☺️
Been an absolute pleasure to TA the class for
@RanjayKrishna
,
@danfei_xu
and
@drfeifei
these past 2 years - super humbled and happy!!
6
2
134
Found some free time to release a library I've been using for the last year to abstract away some PyTorch boilerplate. It's pretty lean and only meant to make my research projects cleaner and more reproducible.
2
17
133
We really take the deep learning field for granted — i can’t read a single inverse kinematics paper without hitting a paywall, it’s quite frustrating.
6
4
115
First, if you enjoyed the video above, we have a live demo that runs MuJoCo in your browser using Javascript and Web Assembly! You can accompany the robot (drag the keys down), or be adversarial and tug at the fingers 🙃
4
14
113
Progress in this field is truly mind boggling. Studying for prelims and slides from last year are out of date 😅
1
5
111
Simulated entirely in MuJoCo, serialized to a brax protobuf which is then rendered with three.js in the browser. cc
@araffin2
@natolambert
@markus_with_k
6
7
110
New blog post: "Representation Matters" How cleverly designing your state and action space can give you orders of magnitude more sample efficiency in imitation learning.
4
21
111
The dedication and attention to detail in the appendix of the RGB Stacking paper by the robotics folks at Deepmind is insane 🤯 A window into the complexities and engineering efforts that come with real-world robot evaluations..
4
15
106
Finally got around to port this over to MuJoCo!
couldn’t wait / was working on docs/tutorials to build simulated tasks so went ahead and did the first step, add the assets in
next up is adding that reorienting fixture, then motion planning solution maybe to generate demos
4
2
53
3
4
99
Reading old robotics theses just makes me realize how much the field is regressing due to hype 🫠
5
4
98
MuJoCo 3.0 has gotten significantly better at simulating deformable materials and it beats NVIDIA Flex and other simulators in this project. Congrats
@TheSmallQuail
!!
Interested in deformable objects, cloth manipulation or simulation engines?
We have evaluated the Sim-to-Real gap in
@GoogleDeepMind
MuJoCo, Bullet,
@nvidia
Flex and
@SofaFramework
SOFA on dynamic and quasi-static cloth manipulation!
Do you want to know more? Scroll down! 🧵
6
16
86
1
11
96
Life update: I'm spending the summer in NYC as a student researcher in Brain, hosted by the wonderful
@andyzengtweets
and
@peteflorence
. Excited to work on and push the frontiers of dexterous manipulation 🤟
4
0
93
I've been working with huge numpy arrays the last few days and it's made me realize how crappy my normal numpy style is. v1 is what I would have done previously because it's more readable. But v2 is giving me an almost 3x speedup which is insane.
3
7
90
Some RoboPianist updates:
1. RL training code is now open-sourced:
2. Pixel observations are now available via a wrapper:
0
10
88
If you're using MuJoCo and want to play with this dataset, I've made a tool that automatically processes OBJ files and generates XML files for you:
Here's obj2mjcf in action using 2 objects from this dataset:
Presenting the open-sourced Scanned Objects dataset, a curated collection of over 1k 3D-scanned common household items that realistically reflect object properties, like mass, resulting in a more realistic simulation. Learn more about this effort →
23
74
260
2
13
88
Dove into a new area at the beginning of the year: dexterous hand manipulation. It's been incredibly gratifying to learn new things on the daily: new simulator (MuJoCo), inverse kinematics (differential & QP), and now motion capture and retargeting (little bit of Blender) 😀
3
2
84
Looking forward to present RoboPianist in Atlanta for
#CoRL23
this week! DMs are open in case you want to talk robotics, MuJoCo or anything else.
• Poster session 3 (4:15-5:00 PM)
• Updated paper:
Introducing 𝗥𝗼𝗯𝗼𝗣𝗶𝗮𝗻𝗶𝘀𝘁 🎹🤖, a new benchmark for high-dimensional robot control! Solving it requires mastering the piano with two anthropomorphic hands.
This has been one year in the making, and I couldn’t be happier to release it today! Some highlights below:
55
236
1K
2
6
84
Our paper has been accepted to ICRA 2020! Looking forward to meeting and chatting with new and old faces there 🙂
2
14
84
It's so inspiring to see OpenAI just push better and better models to their API and actually productionize all these amazing models they've developed over the last few years.
1
1
76
L take. BD was doing the impossible for a few years and pushing the state-of-the-art on many fronts. Now all of a sudden what they did was gimmicky because a few startups started doing teleop and "re-invented" 2015 results to the public eye.
Atlas
@BostonDynamics
is finally doing tasks that are actually useful!! I'm much more excited by this than parkour and backflips in the past. From the first person view, it looks like Atlas is estimating the pose of the object and its own hands (with a digital twin overlay),
67
197
1K
3
4
77
The ViperX 6-DoF arm from
@trossenrobotics
is now available in Menagerie with some initial sysID by the amazing
@the_real_btaba
!
1
3
74
Pre pandemic: We leave real world robot experiments for future work
In pandemic: due to covid-19, we limit our experiments to simulation
Post pandemic: Because of the pandemic, we forgot how to use real robots + takes too much time. As such, we stick to simulation only
4
3
73
Step 1: Don’t optimize for this objective..
1
3
73
I'll be joining X
@Theteamatx
as an AI Resident this summer, working on the Everyday Robot Project. Super excited to make progress on teaching robots to be useful in our messy everyday lives.
0
1
70
Stretch from
@hellorobotinc
is now available in Menagerie 😀 Massive thanks to
@binitshah_
for helping with the release!
1
7
69
Lol, just discovered this while inspecting the OpenAI website's HTML code
4
1
68
Finally got around to moving my RL stack exclusively to dm_env (no more gym) and "I've never felt better. I have more energy. My skin is clearer. My eye sight has improved." 😉
3
2
64
@RogerGrosse
@hardmaru
That’s a sweeping generalization imo. And I don’t think it’s too much asking to do a bit of hard thinking about potential misuse/negative impact of the work you’re submitting. It’s academia as you mentioned, go discuss it with your neighbors studying just that..
1
0
64
Unsupervised learning has been super fun to watch in the last 6 months. MoCo a few months back, then SimCLR last month, and now MoCo v2 borrowing two improvements from SimCLR (data augmentation + projection head) to retake the crown.
2
14
65
We need more tweet threads like this!
Want to make your inference code in PyTorch run faster? Here’s a quick thread on doing exactly that.
1. Replace () with the ✨torch.inference_mode()✨ context manager.
8
360
2K
0
2
65
4:00 AM fun: implemented "Neural Arithmetic Logic Units" in
@PyTorch
. Reproducing each section of the paper so still in progress.
0
16
64
