Excited to share that 𝐋𝐀𝐏𝐀 has won the Best Paper Award at the CoRL 2024 Language and Robot Learning workshop, selected among 75 accepted papers! Both @SeonghyeonYe and I come from NLP backgrounds, where everything is built around tokenization. Drawing inspiration from Bye-Pair Encoding in language models, 𝐋𝐀𝐏𝐀 takes a similar approach by tokenizing actions directly from video data. While BPE is frequency-based, ours is learning-based where we compare the difference between two consecutive frames in videos to determine the atomic action granularity. This allows Vision-Language-Action models to learn effectively from human video data as well as gain stronger x-embodiment transfer—going beyond pre-defined robot actions such as end-effector deltas. As popular request, we’re also releasing the core component behind this success: the VQ-VAE model training code to extract latent actions. This open-source release includes the GENIE model implementation with added enhancements to minimize codebook collapse, allowing you to input ANY type of video—robotic, human, navigation, etc.—and extract latent actions to train visuomotor robot policies. We worked hard on making it easy to use, so do try it out! We believe actions can be a more expressive form of language. Code: