Project · 2026 · completed · Academic
ACT-based real-robot manipulation with LeRobot
An imitation-learning study that trains Action Chunking Transformer policies from demonstrations and evaluates them on a physical SO101 robot arm.
Problem
Learning manipulation directly from demonstrations can replace brittle, hand-programmed motion sequences, but performance on a physical robot depends on more than training loss. The project studied whether an Action Chunking Transformer (ACT) could learn useful pick-and-place behavior with a lightweight setup: one RGB camera, robot-state input, and a SO101 arm. It also explored how camera placement and task complexity affect real-world execution.
Approach
Demonstrations were recorded as synchronized image, robot-state, and action sequences in a LeRobot-compatible format. The state and action vectors each contain six joint-position channels covering the arm and gripper. From one image observation and the current robot state, the ACT policy predicts a chunk of 100 future actions.
The main model combines an ImageNet-pretrained ResNet-18 visual backbone with a Transformer using a 512-dimensional hidden representation, eight attention heads, four encoder layers, and one decoder layer. It was trained with AdamW for 100,000 optimization steps on 50 side-camera demonstrations, then deployed directly on the physical robot.
Experiments and results
| Experiment | Demonstrations | Frames | Physical evaluation | Outcome |
|---|---|---|---|---|
| Side-camera pick-and-place | 50 | 19,515 | 10 runs | 5 complete successes; every successful grasp was placed correctly |
| Overhead-camera pick-and-place | 50 | 20,943 | 3 runs | Evaluation stopped after recurrent self-occlusion during grasping |
| Hanoi-inspired object transfer | 90 | 40,186 | 10 runs | 2 successful transfers in a more complex multi-object setting |
The side-camera policy learned the full approach, grasp, lift, transport, placement, and return sequence, reaching a 50% task success rate across ten physical runs. Failures were concentrated around precise alignment and grasp initiation. In one trial, the policy displaced the cube, repositioned, and later completed the grasp, suggesting limited recovery-like behavior without establishing a general recovery mechanism.
The overhead-view variant is an informative negative result. Although its training loss decreased substantially, the arm repeatedly hid the object at the critical grasping stage. The comparison shows why observability and camera placement must be treated as core parts of a visuomotor learning system rather than as secondary implementation details.
Limitations and takeaway
The experiments were exploratory rather than benchmark-scale: all demonstrations were used for training, the main success estimate is based on ten runs, no camera calibration or controlled viewpoint ablation was performed, and the three variants do not share identical evaluation protocols. The results therefore support a measured conclusion: ACT learned meaningful real-robot manipulation behavior in this setup, but robustness remained sensitive to viewpoint, object visibility, grasp precision, and task complexity.