D-JEPA Aligns Latent World Models with Action Decisions

Introduction

Latent world models usually predict what may happen after an action and select the candidate whose predicted future is closest to a goal. This works only if geometric closeness in the latent space corresponds to practical success. D-JEPA focuses on the cases where that assumption breaks: among a small set of executable candidates, the future that appears closer to the goal can lead to a worse realized outcome than another available option.

The paper calls this a decision-local prediction gap. Rather than treating it as a reason to discard pretrained predictive models, D-JEPA adds a decision-alignment layer that learns how candidate futures should be compared when a control decision is imminent.

Key ideas

• Decision-relevant relations. The method learns relative structure between candidates, not just the quality of each prediction in isolation. The supervision comes from outcomes observed after actions are executed.
• Ordinal evidence. Instead of requiring a fully calibrated scalar value for every future, D-JEPA uses ordering information: which candidate performed better or worse in the relevant decision set.
• Bounded, permutation-equivariant correction. A bounded operator jointly processes goal-relative predictive features and ordinal evidence. Its permutation equivariance means that reordering candidates should not change the underlying decision logic.
• Limited adaptation of the predictor. The approach restricts how the pretrained predictor is changed, aiming to preserve its predictive geometry while correcting it where action ranking matters most.
• JEPA-compatible deployment. The learned decision structure is realized in future representations compatible with JEPA-style planning, allowing native latent-distance planning rather than requiring a wholly separate controller.

Results and implications

The supplied material reports evaluations spanning latent control, manipulation, pretrained action-producing models, physical robots, and autonomous driving. D-JEPA reaches 87.89% success in the PushT confirmation evaluation, reports a 15.04-percentage-point average gain on RoboTwin, and improves physical robot tasks by 17 points. These figures suggest that better control can come not only from more accurate forecasting, but from making comparisons between forecasts more useful for action selection.

The distinction matters in embodied systems. A robot or vehicle typically evaluates a limited set of alternatives under time and compute constraints. If the latent metric ranks those alternatives incorrectly, even a strong predictor may produce poor behavior. D-JEPA offers a way to use execution feedback to repair that local ranking while retaining the infrastructure of a pretrained world model.

The approach is not presented as a universal replacement for predictive modeling. Its results can depend on decision supervision, candidate generation, and the geometry inherited from the base model. Still, it frames an important design principle: a useful world model must represent not only what could happen, but also which predicted future is worth acting on.

Source: Hugging Face Daily Papers