Luce Turns Single Images into Relightable 3D Assets

Introduction

Single-image 3D generation is often judged by whether a rendered object resembles the input from one viewpoint. That standard is not enough for assets intended for games, film, design, or conventional rendering pipelines. A useful asset must also have coherent geometry, reusable material properties, and an appearance that remains plausible when the lighting changes.

The paper Luce: Relightable Gaussians for 3D Asset Generation targets this gap. Rather than producing only a visually convincing reconstruction, Luce aims to generate a 3D asset whose appearance can be decomposed into physically based components and relit after generation.

Key points

• A shared multimodal Gaussian representation. Luce places geometry and PBR information in a voxelized multimodal Gaussian cloud. Dedicated Gaussian primitives are used for separate modalities, including albedo, metallic-roughness, and surface normals.
• A material-aware latent space. A variational autoencoder compresses the joint representation into a unified latent space. This connects shape and appearance before the image-conditioned generation stage, instead of treating material recovery as an independent afterthought.
• Single-image latent generation. A rectified-flow Transformer predicts the 3D latent from one image. It is conditioned on multi-layer features from a pretrained image encoder, combining high-level semantic context with fine spatial information.
• Two practical output paths. The decoded result is a relightable PBR Gaussian asset. When a mesh-based workflow is preferred, Luce can also produce an optional textured mesh with a tangent-space normal map.
• Reported gains in reconstruction and alignment. On Toys4K, the authors report a 28% FID improvement over the strongest baseline. On a newly introduced benchmark of AI-generated images, Luce reaches a CLIP image-alignment score of 0.8519, compared with 0.8299 for the best baseline. The paper also highlights preservation of small details such as text, logos, and inscriptions.

Why it matters

The central contribution is not simply another 3D output format. Luce combines the efficient, appearance-oriented nature of Gaussian representations with explicit PBR modalities that can be consumed by lighting and rendering systems. In principle, this can narrow the gap between a generated view-dependent result and an asset that can be placed into a broader content pipeline.

Its architecture also reflects a useful division of labor. The autoencoder learns a joint geometry-material representation, while the image-conditioned flow model focuses on generating that representation from visual evidence. Multi-layer image features are intended to preserve both object identity and local details, which is particularly relevant for markings and surface patterns.

The available material does not establish the method’s inference cost, category-by-category failure modes, or how its Gaussian and mesh outputs compare in full production workflows. Those questions require the complete paper, implementation, and broader testing. Still, the reported results position Luce as a step toward single-image 3D generation that produces relightable, materially specified assets rather than merely plausible geometry.

Source: Hugging Face Daily Papers