Robots Cannot Wait for Inference: SmoothRL Aligns Online RL with Asynchronous Execution

A physical robot cannot simply pause whenever a large model needs more time to reason. Vision-language-action and world-action models often predict an action chunk, while the robot is already carrying out the previous one. If inference takes longer than the control loop, a pause can destroy accumulated momentum in tasks such as throwing.

Asynchronous execution keeps the robot moving, but it creates a credit-assignment problem for online reinforcement learning. A model may generate 32 actions, yet only some of them will be executed before the next inference result replaces the rest. Treating the entire chunk as an RL training target can therefore reward or penalize actions that never happened in the physical world.

Key ideas

• Split each action chunk by execution status. SmoothRL distinguishes a committed region, an execution region, and a discarded region. Committed actions have already been handed off and cannot be changed; the execution region contains actions the robot will actually perform; discarded actions are overwritten before execution.
• Train on physical reality. Policy gradients are allowed to pass only through the execution region. This makes the learning signal correspond to the robot’s actual trajectory rather than to the model’s full prediction.
• Reproduce deployment timing during rollout. Inference and robot control run in parallel during training, and the replay buffer records the resulting asynchronous timing. The approach is summarized as “Reinforce in Deployment.”
• Use residual policy updates. The reported implementation starts from task-adapted π0.5 policies and uses a lightweight TD3-style actor-critic to predict residual corrections in the original action space.

The experiments ran on Astribot’s S1 robot. The control loop operated at 30 Hz, while new inference requests arrived at 5 Hz. Under the stated latency budget, only part of each predicted chunk became executable, illustrating why synchronous training would misrepresent deployment.

Across three real-robot tasks, the reported success rate for dynamic throwing rose from 39% to 94%. Pen-capping improved from 8% to 83%, while parcel opening increased from 30% to 90%. The tasks test different forms of difficulty: maintaining speed before release, aligning two arms within a small tolerance, and inserting a roughly 1 mm blade into a narrow seam. The opening task also showed a non-monotonic learning curve, with performance falling before recovering, a reminder that online exploration in the real world is not guaranteed to improve smoothly.

Why it matters

SmoothRL is not presented as a way to teach a robot a task from scratch. Its role is closer to deployment-time post-training: start with a policy that broadly knows what to do, then use real outcomes to correct repeatable errors. The reported examples include incorrect release speeds, a persistent leftward bias during opening, and insufficient adaptation to cap positions. The team also reported a 52% reduction in endpoint acceleration RMS and a 47% reduction in jerk during one autonomous throwing rollout after online RL.

There are clear limitations. The current setup uses sparse success/failure rewards and allows operator intervention when needed. The residual policy is constrained by the frozen base policy, and every chunk-level inference must remain within a predefined latency budget. If the initial behavior is far from the target, local corrections may not be enough. SmoothRL therefore points to a practical layer between pretrained robot competence and reliable physical deployment: learning from what the robot truly did, not merely from what the model intended.

Source: QbitAI