Steering the robot like a puppet

The most direct way to teach a robot a task is to do it through the robot. A person steers the robot’s arms remotely, which is called teleoperation, while the robot records what its cameras saw and how its joints moved. A learning program studies those recordings and tries to copy them.

In 2023, a team of researchers showed how far this can go with modest hardware. Their two-armed system, called ALOHA, was built on a budget of about $20,000. After 50 demonstrations per task, about 10 minutes of recordings, the robot learned fiddly jobs such as opening a clear plastic sauce cup and slotting a battery, succeeding 80 to 90 percent of the time.

Pooling what many robots learned

Fifty demonstrations is fine for one task. A robot meant for many jobs needs far more. So labs have started pooling data.

The Open X-Embodiment project, published in 2023, gathered recordings from 22 different robots across 21 institutions, covering 527 skills. The team found that a model trained on the pooled data could improve several robots at once, by drawing on experience from other machines.

Robots that read

Many recent systems borrow from chatbots. A vision-language-action model starts from an AI that already understands pictures and words, learned from the web. It is then trained further on robot recordings, so that it learns to answer with movements as well as text.

RT-2, a model described in 2023, did this by writing robot movements as a short row of numbers, the same way it writes words. Because it had learned from the web, it could follow requests it never saw in robot training, such as “pick up the extinct animal”. π0, the laundry-folding model from lesson 1, takes a similar approach. The company says it was trained partly on its own recordings from 8 different robots.

From lab to factory floor

On 20 October 2026, the farm machinery maker AGCO and Agility Robotics announced a plan to test Agility’s Digit humanoid at AGCO’s plant in Hesston, Kansas. According to the announcement, Digit will start by sorting and stacking parts and putting bins into storage. The pilot is expected to begin in 2027, so there are no results yet.

What to watch for

When a robot demo goes viral, read the small print. Is the video sped up? Look for a playback speed in the caption. Was a person steering it, or was it acting on its own? How many tries came before the one you see? Some teams, like Physical Intelligence, show failures alongside successes, which is a good sign. If the source doesn’t say, treat the clip as a best case.

In one sentence: robots now learn by copying people who steer them, pool those recordings across many machines, and borrow web knowledge from chatbot-style models.