论文标题
RGBD2：使用RGBD扩散模型通过增量视图介入的生成场景合成
RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models
论文作者
论文摘要
我们解决了从稀疏的RGBD视图观测值集中恢复基础场景几何形状和颜色的挑战。在这项工作中，我们提出了一种称为RGBD $^2 $的新解决方案，该解决方案沿相机轨迹依次生成新颖的RGBD视图，场景几何形状仅仅是这些视图的融合结果。更具体地说，我们维护一个用于呈现新RGBD视图的中间表面网格，随后通过indpainting网络变得完整。后来将每个渲染的RGBD视图反射为部分表面，并补充到中间网格中。中间网格和相机投影的使用有助于解决多视图矛盾的棘手问题。我们实际上将RGBD插入网络实现为多功能RGBD扩散模型，该模型以前用于2D生成建模。我们对其反向扩散过程进行了修改，以实现我们的使用。我们从稀疏的RGBD输入中评估了3D场景合成任务的方法；扫描仪数据集的广泛实验证明了我们的方法优于现有方法。项目页面：https：//jblei.site/proj/rgbd-diffusion。
We address the challenge of recovering an underlying scene geometry and colors from a sparse set of RGBD view observations. In this work, we present a new solution termed RGBD$^2$ that sequentially generates novel RGBD views along a camera trajectory, and the scene geometry is simply the fusion result of these views. More specifically, we maintain an intermediate surface mesh used for rendering new RGBD views, which subsequently becomes complete by an inpainting network; each rendered RGBD view is later back-projected as a partial surface and is supplemented into the intermediate mesh. The use of intermediate mesh and camera projection helps solve the tough problem of multi-view inconsistency. We practically implement the RGBD inpainting network as a versatile RGBD diffusion model, which is previously used for 2D generative modeling; we make a modification to its reverse diffusion process to enable our use. We evaluate our approach on the task of 3D scene synthesis from sparse RGBD inputs; extensive experiments on the ScanNet dataset demonstrate the superiority of our approach over existing ones. Project page: https://jblei.site/proj/rgbd-diffusion.
