Roll out and learn concurrently
Robot interaction and policy optimization overlap. Actor parameters are synchronized after each update group.
REAL-WORLD ROBOT LEARNING
RAPolicyStable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvement
1 Institute of Trustworthy Embodied AI, Fudan University
2 Shanghai Key Laboratory of Multimodal Embodied AI
3 Singapore Management University
From real-world experience to reliable action.Project overview · original playback speed
THE IDEA
Keep the robot moving.
Keep learning grounded.
RAPolicy lets a vision-language-action model collect experience and learn at the same time. Both the critic and actor learn from recorded behavior, grounding policy improvement in what the robot has actually experienced.
Starting from just 10 demonstrations per single task, RAPolicy adapts through real-world interaction within 1–2 hours, with fewer human interventions.
86.3%
Average single-task success
52 → 88%
Joint five-task success
1–2 hours
Online training budgets
01 / METHOD
Asynchronous execution.
Replay-anchored improvement.
Reuse successes, failures, and human corrections while the deployed policy continues to collect new experience.
Anchor value learning in replay
Chunk-level critics construct Bellman targets without sampling next actions from the changing policy.
Improve the one-step actor
Stored rollout latents pair with recorded actions for advantage-weighted conditional-likelihood updates, without critic action gradients.
02 / SINGLE-TASK ADAPTATION
Learning on the real robot.
Four tasks span precision, contact-rich manipulation, and visual understanding. Full supplied online recordings are shown at 10× speed, with audio removed.
Plug Charger
10× speedPrecision insertion Final evaluation: 100%
Pick Banana
10× speedPick and place Final evaluation: 95%
Stack Blocks
10× speedPrecise stacking Final evaluation: 70%
Wipe Whiteboard
10× speedContact-rich manipulation Final evaluation: 80%
Higher success under the same online budgets
Final policies evaluated over 20 trials per task. Values are success rates (%); “—” indicates a result not reported.
| Task | SFT | HIL-SERL | ALOE | EXPO-FT | RAPolicy |
|---|---|---|---|---|---|
| Plug Charger | 5 | 100 | 30 | 80 | 100 |
| Pick Banana | 15 | — | 55 | 80 | 95 |
| Stack Blocks | 0 | — | 5 | 30 | 70 |
| Wipe Whiteboard | 10 | — | 50 | 10 | 80 |
03 / JOINT MULTI-TASK ADAPTATION
One policy. Five tasks.
Continuous improvement.
With 30 initial demonstrations per task and a two-hour online budget, overall success rises from 52% to 88%, while already reliable skills remain strong.
RAPolicy · Online learning
10× speed
Evaluation after adaptation
Ten trials per task, across five tasks. The recordings below are presented at 10× speed.
Initial SFT
52%26 / 50 successful trials
HG-DAgger
70%35 / 50 successful trials
RAPolicy
88%44 / 50 successful trials
| Task | SFT | HG-DAgger | RAPolicy |
|---|---|---|---|
| Lemon → basket | 40% | 70% | 90% |
| Yellow block → basket | 90% | 90% | 100% |
| Green block → basket | 100% | 100% | 100% |
| Yellow block on green | 20% | 30% | 90% |
| Green block on yellow | 10% | 60% | 60% |
04 / ABLATIONS
What makes the updates work?
On Plug Charger, ablations isolate the one-step policy, replay-anchored value targets, and stored-latent actor supervision. The full method reaches 100% rollout success within 30 minutes in this experiment.
Explore the implementation ↗
ABSTRACT
RAPolicy
Online post-training of vision-language-action (VLA) models requires efficient use of robot interaction and reliable policy improvement from continually collected experience. We propose asynchronous Replay-Anchored Policy improvement (RAPolicy), a framework that performs rollout and learning concurrently while grounding both critic and actor updates in replayed behavior. The critic learns chunk-level values from recorded actions and constructs Bellman targets without predicting next actions, reducing computation and dependence on action-value estimates outside replay coverage. The one-step flow actor reuses the initial noise stored during rollout and learns through advantage-weighted conditional likelihood, directly supervising the action mapping used for execution. We evaluate RAPolicy across four single-task settings and one joint five-task setting in the real world, with online training budgets of only 1–2 hours. Starting from policies fine-tuned on just 10 demonstrations per task, RAPolicy rapidly adapts to new single tasks and achieves an average 86.3% success rate. In the joint multi-task setting, RAPolicy improves overall success rate from 52% to 88% while preserving performance on already reliable tasks and improving weaker capabilities. Overall, RAPolicy substantially outperforms the baselines in aggregate task success while requiring fewer human interventions, demonstrating stable policy improvement and high online training efficiency.