Our robot dog can go up and down stairs, walk on stepping stones where even a single bad foot placement would lead to a disastrous fall, and rough terrain. All with just a single onboard RGBD camera & no maps.
We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language. Check out our robot walking in San Francisco (Ilija Radosavovic et al)
Another success of sim-to-real for training robot policies! This task, using two multi-fingered hands, requires considerable dexterity, and is hopefully representative of other household tasks that we wish to solve in the future.
Achieving bimanual dexterity with RL + Sim2Real!
TLDR - We train two robot hands to twist bottle lids using deep RL followed by sim-to-real. A single policy trained with simple simulated bottles can generalize to drastically different real-world objects.
Super excited to announce our new work: Synthesizing Moving People with 3D Control (3DHM)💡
Why is 3DHM unique?
With 3D Control, 3DHM can animate a 𝗿𝗮𝗻𝗱𝗼𝗺 human photo with 𝗮𝗻𝘆 poses in a 𝟯𝟲𝟬-𝗱𝗲𝗴𝗿𝗲𝗲 camera view and 𝗮𝗻𝘆 camera azimuths from 𝗮𝗻𝘆 video!
Just to be clear, this is a classic problem in computer vision. The novelty is in the high quality of the 3D reconstructions, even in the presence of occlusions and for unusual poses.
@ToruO_O
There have been impressive recent results using tele-op trajectories for training robot policies. These are typically for hands that are parallel jaw grippers; work led by
@ToruO_O
in our lab at BAIR shows that one can do this now for multifingered hands with vision and touch.
@jane_h_wu
@geopavlakos
@georgiagkioxari
Exciting results on the ability to 3dfy hands and objects given only a single image. This could pave the way for training robots to perform manipulation tasks by watching Internet videos.