HarmoHOI: Jointly Generating Multi-view HOI Appearance and 3D Motion

Intro

Hand-object interaction synthesis is a practical but difficult problem for animation production and embodied AI. The challenge is not only to make the generated hands and objects look realistic, but also to keep every view synchronized and geometrically consistent. HarmoHOI argues that this requires more than strong video priors: it needs globally aligned 3D geometry and motion inside the generation loop.

Key points

• A unified diffusion design: the model jointly generates synchronized multi-view RGB videos and globally aligned 3D point tracks.
• Pseudo-video representation: 3D tracks are converted into a video-like format so they can live closer to the latent space of 2D foundation models, reducing the domain gap.
• Geometry-first refinement: Global Motion Aligning Diffusion improves coarse trajectories into metric-scale, globally aligned 3D motion.
• Co-evolving appearance and motion: 2D appearance and 3D motion are refined together during denoising instead of being handled separately.
• Data curriculum learning: because multi-view HOI data is scarce, the method transfers generic priors from single-view data into the synchronized multi-view setting.

Why it matters

The main value of HarmoHOI is conceptual as much as technical. Multi-view consistency in HOI is often broken by occlusions, fast hand motion, and viewpoint changes. A pure image or video generator can produce plausible frames, yet still fail to preserve the same hand pose or object alignment across cameras. By making 3D motion a first-class target, the method gives the model an explicit spatial anchor.

This makes the work relevant to several downstream areas. For animation, it can reduce the manual effort needed to match hand-object actions across cameras. For embodied AI, better synthetic HOI data may help train systems that need to understand grasping, placing, handing over, and other fine-grained manipulations. More broadly, it shows how diffusion models can be extended beyond appearance generation toward structured, geometry-aware synthesis.

Editorial take

HarmoHOI is interesting because it does not treat multi-view consistency as a cosmetic upgrade. It treats geometry as the backbone of synthesis. If the approach scales to more object categories, richer actions, and broader real-world scenes, it could become a useful building block for both generative graphics and embodied intelligence.

Source: Hugging Face Daily Papers