Back to articles

Direct3D-V3 Generates 3D Geometry Directly in Voxel Space

3 min read

Many 3D generation systems follow a two-stage recipe: a variational autoencoder (VAE) compresses geometry into a latent representation, and a generator learns to model that representation. Direct3D-V3 challenges this convention. Its diffusion transformer (DiT) operates directly on signed-distance-field (SDF) voxel grids, both during training and sampling, rather than generating a compressed proxy that must later be decoded.

How the approach works

An SDF stores each voxel’s distance from a shape’s surface, providing a representation from which geometry can be extracted. Modeling SDF values directly places the generator’s objective in the voxel domain and removes the VAE reconstruction stage. The trade-off is that the model must learn geometric structure without relying on a compact latent representation, including how neighboring values combine into a coherent surface.

Direct3D-V3 introduces several mechanisms to address that challenge:

  • Implicit manifold regularization: A constraint over neighboring voxels is designed to reduce non-watertight surfaces.
  • Mesh-based normal supervision: The predicted SDF is converted into a mesh with Marching Cubes. Rendered normal losses then provide supervision tied to the orientation of the predicted surface.
  • Camera prediction: The model also predicts the camera parameters associated with the conditioning image. This auxiliary task is intended to encourage a stronger mapping between image observations and 3D geometry.

The authors say the latent-free design produces meshes with substantially richer local detail and avoids a reconstruction bottleneck. They also describe an extension to generative reconstruction: diffusion is interleaved with normal-guided refinement, aiming to align the result with the observed view while allowing the generative prior to complete unseen regions.

Those claims should be read in context. The supplied paper summary does not include benchmark results, computational costs, voxel resolution, or experimental comparisons with latent-based systems. It therefore supports describing the method and the authors’ reported outcome, but not quantifying how much it improves on alternatives or whether the direct representation is more efficient in practice.

Why it matters

The work highlights a central design choice in 3D generation. Latent representations can make modeling more manageable, but a VAE’s encoding and reconstruction may limit the fidelity of the final geometry. Direct voxel modeling removes that conversion step, while putting greater pressure on the generator to maintain coherent surfaces. The manifold constraint, normal losses, and camera-prediction task are all attempts to make that direct route viable.

If the approach proves robust across object types and practical resolutions, it could offer an alternative for image-to-3D generation and single-view reconstruction. Its broader significance is currently methodological: rather than improving the latent representation, Direct3D-V3 asks whether a diffusion model can learn geometry in its native form. Evaluating the trade-off will require fuller experimental evidence on quality, scaling, and cost.

Source: Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles

CCTest · Blog
Posterior Matching Distillation Targets Faster, More Diverse Diffusion
Diffusion Models
cctest.ai
Diffusion Models

Posterior Matching Distillation Targets Faster, More Diverse Diffusion

Posterior Matching Distillation (PMD) aims to compress diffusion sampling without reducing the student to only the teacher’s dominant modes. The paper reports leading FID among four-step methods on SDXL and FLUX.2, though a comment on the paper page flags an inconsistency in the FLUX.2 figures.

Read more