From Human Demonstrations to Robot Designs

This short post is about a weekend project I did, inspired by REAL Lab’s research on data collection devices (iPhUMI) and co-optimization of hardware, controls, and learning (Transformer Transformer).

The iPhUMI system consists of a handheld gripper with an iPhone attached used to record humans demonstrating tasks along with a pipeline to convert these demos into a format suitable for training robots. I started by putting together my own iPhUMI device using their open-source documentation, modifying the mount attachment to accommodate a different iPhone from the one used originally. Flow state was achieved during the sub-30-minute assembly!

The setup was smooth, and I recorded myself using the gripper to pick up my safety glasses from the ground and placed them on my standing desk. I repeated this five times, a quantity mainly grounded by vibes.

The next stage involved using the end effector trajectory from the demos to generate robot designs optimized for this set of motions using the Transformer Transformer system. The core idea behind this system is an attempt to answer the question, “What is the best robot embodiment for a given manipulation task?“.

Rather than “make a robot from scratch” (an unimaginably complex optimization), I constrained the generation to a specific design space. I chose the ViperX (robot arm) kinematic design space because it was the simplest released setup that would allow me to go through and understand the full demonstrate, generate, and validate pipeline. It also did not make much sense to test a bimanual robot for my single-gripper trajectories.

After extending the pipeline to support single-arm demos, I ran the released ViperX checkpoint on my safety-glasses pickup demos. My dusty Asus ROG G14 was past its prime, and it struggled to run the Diffusion Transformer. This problem was solved for 49 cents an hour when I rented my first ever GPU: an Nvidia A40.

I tested two generation strategies. In the individual condition, Transformer Transformer generated a separate robot for each demonstration. In the joint condition, it generated one shared robot conditioned on all demos.

Now, five reps of the same task and one robot type are not enough data to draw conclusions. But in this small test, only one of the five individual designs completed its assigned simulated trajectory, while the shared design completed all five.

I’ll continue to update this post as I go deeper and get better insights, but a huge thanks to Austin Patel, Huy Ha, and their labs for open sourcing this and having superb documentation.