How can robot hands learn to do kitchen tasks from safe, continual real-world interaction? In our new CoRL 2023 paper DEFT, we learn policy priors from human videos and use our soft robot hand to safely adapt the policy with real world practice. 1/

In 270 days, the Department of Commerce will determine whether they will allow open-weights or not. if you support open model weights and want something actionable to do, then figure out how to lobby your opinion to them.

I had a wonderful time working w/ Konwoo Kim on our #NeurIPS2023 paper: We derive a simple+scalable algorithm for learning safety constraints from demonstrations. Using it, we solve antmaze w/o the agent *ever* interacting with the walls! Details at gokul.dev/icl/ [1/n]

3 mo. ago we released the Open X-Embodiment dataset, today we’re doing the next step: Introducing Octo 🐙, a generalist robot policy, trained on 800k robot trajectories, stronger than RT-1X, flexible observation + action spaces, fully open source! 💻: octo-models.github.io /🧵

[1/6] The first step to 3D is getting camera poses. But typical pipelines struggle in sparse-view setups bc of texture-less surfaces, symmetries, or insufficient overlap Our #3DV2024 paper RelPose++ uses a probabilistic energy-based model to get accurate 6D poses from <10 views!

Transformers have revolutionized LLMs, but we found that deep state space models (SSMs) do much better on noisy sensory data. Building on this, we introduce Hierarchical SSM (HiSS), outperforming LSTM, Transformer as well as S4 and Mamba on a range of sensory prediction tasks!🧵

Our #ICLR2024 (Oral) paper parameterizes cameras as bundles of rays for sparse-view pose estimation. We train a diffusion model to predict this representation which can be seamlessly converted to classic camera representations using least-squares! [1/N] jasonyzhang.com/RayDiffusion

After two years, it is my pleasure to introduce “DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset” DROID is the most diverse robotic interaction dataset ever released, including 385 hours of data collected across 564 diverse scenes in real-world households and offices

This was a fun and really important collaboration to participate in. Shoutout to Alexander Khazatsky and Karl Pertsch for their leadership ❤️ Can’t wait to see all the amazing projects that this data will spawn 🤖

It took a lot of work over multiple years but I'm proud to be able to finally share my first #ICML2024 paper: a new, fundamental framework for designing efficient inverse RL algorithms that we're in the process of scaling to real robots: gokul.dev/hyper/! 1/

With HRP, we are able to use human video data to improve almost any visual representation for downstream robot tasks! w/ Mohan Kumar Srirama Sudeep Dasari Abhinav Gupta

We are demoing our LEAP Hand teleoperation system today (Thursday) at #rss2024 with MANUS™ in van Hasseltzaal (the demo room behind the poster rooms in the 3rd floor). Come try and teleoperate our low-cost easy-to-build dexterous robots hands!

🤔 Can we train one policy to control a wide range of robots, from drones to quadrupeds, navigators to bimanual manipulators, and more? 🦾Introducing CrossFormer: a single policy that can perform manipulation, navigation, aviation, and locomotion: crossformer-model.github.io

Very excited to announce that in Fall 2025, I will be starting as an Assistant Professor at the Kahlert School of Computing at the University of Utah University of Utah.