Motion Beyond Morphology: A New Take on Cross-Category Video Motion Transfer

Introduction

Video motion transfer seeks to animate a target object using the dynamics observed in a reference video. Many existing methods implicitly or explicitly depend on structural correspondence: a reference body part, joint, or local region should map to a similar component in the target. This assumption works best when the two objects share comparable morphology, but it becomes ambiguous when the source and target differ in shape, articulation, or deformation mechanism.

The paper Motion Beyond Morphology: Bootstrapping Cross-Category Motion Transfer from Abstract Motion Representations, featured on Hugging Face Daily Papers, tackles exactly this issue. Its central idea is to move beyond fixed morphology and focus on dynamics that remain meaningful across categories.

Key ideas

• Abstract motion instead of rigid correspondence: Rather than forcing a one-to-one structural mapping, MBM aims to capture transferable motion cues at multiple levels. These may include temporal rhythm, direction, magnitude, and broader patterns of change that can survive a category shift.
• A two-stage pipeline: In Stage I, the method learns complementary abstract motion views at different granularities and uses them to bootstrap cross-category video pairs. These pairs provide supervision for motion transfer across diverse morphologies. In Stage II, the model internalizes this supervision into direct reference-video-conditioned generation, so inference no longer requires an explicit motion extraction step.
• Broader evaluation setting: The authors introduce OpenVMT-Dataset and OpenVMT-Bench for training and evaluating both image-conditioned and text-conditioned motion transfer. The benchmark organizes category gaps into Same, Near, and Far settings, making it possible to test not only similar-object transfer but also more difficult cross-category cases.
• Two evaluation priorities: The work emphasizes both motion fidelity and target preservation. In practical terms, the generated video should follow the reference dynamics while still maintaining the identity, appearance, and semantics of the target object.

Why it matters

The paper points to an important direction for controllable video generation. If motion transfer only works between structurally similar objects, its creative and production value remains limited. Cross-category transfer could enable more flexible animation workflows, richer visual effects, character motion reuse, and open-ended video editing where the reference and target need not belong to the same class.

At the same time, the problem is inherently difficult. A motion pattern does not have a single universal realization: different objects may express comparable dynamics through different physical or visual mechanisms. A model must therefore preserve the reference motion without violating the target’s morphology. MBM’s use of abstract motion representations and bootstrapped cross-category supervision is one proposed way to balance these demands.

The dataset and benchmark are planned for release upon acceptance. If they cover sufficiently diverse morphologies and category gaps, they could become useful infrastructure for evaluating future work in video motion transfer.

Source: Hugging Face Daily Papers