3DarkFusion: 3D-Aware RGB-NIR Imaging for Seeing in the Dark

Introduction

Low-light imaging remains difficult because conventional RGB sensors can degrade rapidly when illumination is scarce. Images captured in the dark often contain heavy noise, weak color information, and unstable details. Near-infrared, or NIR, sensing can provide more reliable structural cues under such conditions, which is why RGB-NIR fusion has become an active direction in computational photography and machine vision.

However, many existing methods depend on carefully prepared training pairs, especially noisy RGB inputs matched with clean RGB references. That requirement is hard to satisfy in real dark scenes: collecting a clean reference of the same moment, viewpoint, and scene configuration is often impractical. The paper “Toward Robust and 3D-Aware RGB-NIR Imaging in the Dark” proposes a different path through a system called 3DarkFusion.

Key ideas

• No clean RGB supervision required: 3DarkFusion is designed to avoid the need for clean RGB images as direct supervision, reducing the data collection burden.
• Fusion in 3D space: Instead of treating RGB-NIR enhancement as a purely 2D image translation problem, the method introduces 3D-aware neural modeling and uses neural rendering to combine information across observations.
• Use of NIR structure cues: NIR data can remain informative when RGB observations are extremely noisy, helping the model infer scene structure and recover more plausible color images.
• Robustness across noise levels: The authors state that the model generalizes under different noise conditions, which is crucial for real-world low-light applications.
• Evaluation on synthetic and real data: The paper reports extensive experiments on both simulated and real captures, with results showing advantages over prior approaches.

Why it matters

The most important shift here is conceptual. 3DarkFusion frames low-light RGB-NIR imaging as a 3D-aware reconstruction and rendering problem rather than a supervised cleanup task that depends on ideal reference images. If this direction proves practical, it could help systems that need visual perception in darkness, including night photography, surveillance, autonomous sensing, robotics, and mobile vision hardware equipped with NIR capabilities.

The approach also points to a broader trend: using geometry-aware neural representations to compensate for missing or unreliable supervision. In difficult imaging conditions, the model can exploit consistency in 3D space and the complementary nature of RGB and NIR signals, rather than relying solely on paired datasets.

Open questions remain, including computational cost, capture requirements, calibration between RGB and NIR sensors, and behavior in dynamic scenes. Still, the paper offers a compelling research direction for robust imaging in severe low-light environments, and the availability of code should make it easier for the community to evaluate and extend the idea.

Source: Hugging Face Daily Papers