MonoDiffSplat

Iterative Single-View 3D Reconstruction via Plane-Guided Depth Refinement and Video Diffusion

Sam (Seyed M.) Hosseini
York University
Repository

Abstract

MonoDiffSplat is a single-view-first 3D reconstruction pipeline that also supports sparse multi-view inputs. Built on G4Splat, Depth Anything 3, See3D, and 2D Gaussian Splatting, it bootstraps depth and plane structure from input views, trains an initial 2D Gaussian model, then runs up to three iterative rounds of novel-view rendering, See3D inpainting, base-cloud-anchored depth refinement, point-cloud quality control, and resumed Gaussian training.

Key extensions over G4Splat: Depth Anything 3 initialization for single-view and sparse inputs, stage-to-stage chaining of plane models and base clouds, structured coverage-driven view selection, and delta geometry injection with floater cleanup during Gaussian retraining. This repository documents and implements the iterative single-view and low-view extensions; it is not a standalone benchmark claim.

Rendered frame from single view reconstruction

Single input image

Generated scene

Generated scene

Rendered frame from single view reconstruction

Single input image

Generated scene

Generated scene

Rendered frame from single view reconstruction

Single input image

Generated scene

Generated scene

Figure 1: Single-view reconstruction examples.

Method Overview

The pipeline has three phases:

  • Bootstrap. Depth Anything 3 produces monocular depth with global alignment. Geometry-branch normals and SAM masks drive plane extraction, plane-aware depth refinement runs, and the first 2D Gaussian model trains.
  • Iterative See3D refinement (1–3 rounds, default 1). Each round renders novel views, runs See3D inpainting with feathered visible-region merge, anchors new depths against the previous round's unified point cloud while carrying forward fitted plane models, exports an aligned point cloud, and resumes Gaussian training with delta injection and floater cleanup.
  • Finalization. Optional adaptive-TSDF tetra mesh extraction and held-out view evaluation.
MonoDiffSplat pipeline diagram

Figure 2: Pipeline summary. Bootstrap builds the first depth-refined point cloud and Gaussian model from sparse RGB inputs. The iterative phase then repeats one to three times: render novel views, inpaint with See3D, refine depth against the previous round's geometry, export QC point clouds, resume Gaussian training, and re-render.

Geometric Consistency

Depth estimates provide geometric supervision throughout refinement and training, reducing floaters, improving planar alignment, and stabilizing reconstructions from limited input views.

RGB image
RGB Image
Depth map
Depth Map

Outdoor scene

RGB image
RGB Image
Depth map
Depth Map

Indoor scene

Extensions Over G4Splat

  • Depth Anything 3 bootstrap. Initial monocular depth estimation with global alignment, including single-view runs, replacing MASt3R-based initialization for sparse input regimes.
  • Base-cloud-anchored depth refinement. See3D depths are aligned to the previous round's QC point cloud via per-segment scale fitting, with hard acceptance gates and an adaptive view-rescue floor. Inherited plane models are matched first, with boundary-anchored linear mono fits as a weak-plane fallback.
  • Geometry-branch normals. Plane extraction on inpainted views uses normals computed from raw mono depth rather than from GS-warp-aligned depth, avoiding projective distortion on regions with sparse Gaussian coverage.
  • Structured view selection. Novel views are drawn from a fixed trajectory mix (parallax, dolly, orbit, top-down, plane-guided), filtered by visibility and coverage, and feather-merged with input-view backward warps to reduce full-frame diffusion drift.
  • QC-gated geometry injection and floater cleanup. Unified point cloud export applies multi-view consistency, plane-extension snapping, and an adaptive inlier gate before injection. Resumed Gaussian training adds depth-prune grace periods, post-densify shard cleanup, and an opacity-gated anisotropy pass.

Acknowledgments

Built on G4Splat (Ni et al., 2025). Additional dependencies: Depth Anything 3, See3D, 2DGS, Segment Anything, GOF. Credit for the base G4Splat formulation remains with the original authors; MonoDiffSplat documents and implements the iterative single-view extensions described in this repository.