InstantHMR Brings Real-Time 3D Human Mesh Recovery to Phones

Introduction

Recovering a three-dimensional human pose and surface mesh from a single image is a useful capability for AR/VR, robotics, biomechanics, fitness applications, and interactive games. Yet deploying such systems on a phone is difficult. Mobile hardware has a much tighter compute and power budget than a desktop GPU, while real-time applications require consistently low latency. Existing approaches often favor either reconstruction quality or execution speed.

InstantHMR is designed around that deployment constraint. It is a family of open single-image regressors that directly predict the parameters needed to recover a human body mesh. The method uses the Momentum Human Rig (MHR) parametric representation and is trained exclusively with publicly available annotations from the SAM 3D Body dataset.

Key points

• Mobile-oriented inference: On the NPU of a Samsung Galaxy S23, InstantHMR reconstructs a 6,890-vertex body mesh for one person in 5.5 ms.
• Fast desktop execution: The reported latency is 0.98 ms per person on an NVIDIA RTX 4070 GPU.
• Competitive accuracy: The model reaches PA-MPJPE values of 40.0 mm on 3DPW, 48.8 mm on EMDB, and 41.7 mm on RICH.
• More controlled comparisons: The authors re-evaluate 11 state-of-the-art methods under a unified protocol aligned with closely related MHR-based systems. The comparison also includes Meta’s official MHR-to-SMPL conversion.
• Broader deployment evidence: Latency is measured across eight targets, ranging from desktop TensorRT to two phone NPUs, and both models and implementation are released publicly.

Why it matters

The main contribution of InstantHMR is not simply a faster benchmark number. It treats mobile deployment as a first-class design objective for human mesh recovery. According to the reported comparison, InstantHMR is about 12 times faster on mobile than the only mobile-compatible method evaluated that slightly exceeds it in accuracy. That difference could determine whether a system can process a live camera stream locally or must rely on a remote service or a more powerful computer.

Local inference can also be useful for product design. Fitness feedback, avatar control, AR interaction, and on-device body analysis all benefit from low latency and reduced dependence on network connectivity. The open release may further lower the barrier for developers who need to adapt a human mesh model to a specific mobile runtime.

The results should still be read within their stated scope. Single-image reconstruction remains sensitive to occlusion, viewpoint, depth ambiguity, and clothing. The supplied material focuses on single-frame latency and benchmark accuracy; it does not establish performance for temporally stable video tracking, crowded scenes, or unusually difficult poses. Differences between body representations, including MHR and SMPL, also make evaluation protocols important when comparing systems.

Overall, InstantHMR represents a practical shift toward deployable 3D vision. By reporting latency across real hardware, using a unified evaluation setup, and releasing code and models, the project addresses the gap between an accurate research model and a component that can actually run on a phone.

Source: Hugging Face Daily Papers