DRACO Brings Fine-Grained Credit Assignment to Long-Horizon Agents

Why trajectory-level rewards are not enough

Reinforcement learning is relatively straightforward when a task has a programmatic checker: the environment can determine whether the final result is correct. Long-horizon agents are different. They may need to plan, call tools, recover from intermediate errors, and interact with an environment over dozens of steps, while many domains offer no executable success signal.

A common fallback is to ask for a rubric-based evaluation after the trajectory is complete. This supplies a useful judgment, but a single scalar still gives weak guidance. If every action receives essentially the same return, the policy has little information about which decisions actually produced a good or bad outcome.

DRACO, or Distributing Rubric-based Advantage for Credit Optimization, addresses this mismatch. The paper studies the outcome-blind setting, where ground-truth success labels and verifiers are unavailable during training.

How DRACO works

• Dynamic rubrics: Rather than relying on a fixed evaluation template, DRACO generates multi-criteria rubrics during training. The rubrics can track the policy as its capabilities evolve.
• Trajectory-level scoring: Once an episode is complete, the trajectory is scored against the generated criteria.
• Step-level redistribution: The evaluation is then assigned to the steps responsible for the annotated rubric items, producing differentiated advantages for policy optimization.
• Closed-form attribution: The redistribution is computed directly. DRACO does not introduce a separately trained attribution or credit-estimation module.

The central idea is not merely to create more reward values. It is to connect each rubric judgment with the actions that are relevant to it. This prevents a long trajectory from being treated as an undifferentiated block and gives GRPO a more targeted learning signal.

Results and broader implications

On AppWorld, DRACO improves over the base model by 15.9 points and exceeds GRPO trained with a sparse ground-truth reward by 5.3 points. The method itself does not use verifiers. On out-of-domain Tau-Bench, it gains 5.3 points over the base model without access to a frontier judge, outperforming both ground-truth-reward training and other rubric-based settings reported in the paper.

These findings suggest that the challenge in long-horizon agent reinforcement learning is not simply the absence of rewards. Credit must also arrive at the right locations in the trajectory. Dynamic rubrics provide a structured way to evaluate behavior, while redistribution turns that evaluation into a signal better suited to policy updates.

The approach still depends on the quality and consistency of rubric evaluation, and its step associations must be meaningful. It therefore does not remove the risks of imperfect or subjective judging. Its contribution is more specific: it offers a lightweight bridge between coarse outcome assessment and fine-grained policy learning, without requiring a new learned attribution system. For tool-use and other multi-step agent domains, DRACO provides a practical direction for training when conventional verifiers are unavailable.

Source: Hugging Face Daily Papers