Visitors:

Point-NeRF: Point-based Neural Radiance Fields

(CVPR 2022 Oral 🤩🤯🔥)

A high performance novel neural radiance representation.


Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu
Kalyan Sunkavalli , Ulrich Neumann

Paper Code

Abstract


Volumetric neural rendering methods like NeRF generate high-quality view synthesis results but are optimized per-scene leading to prohibitive reconstruction time. On the other hand, deep multi-view stereo methods can quickly reconstruct scene geometry via direct network inference. Point-NeRF combines the advantages of these two approaches by using neural 3D point clouds, with associated neural features, to model a radiance field. Point-NeRF can be rendered efficiently by aggregating neural point features near scene surfaces, in a ray marching-based rendering pipeline. Moreover, Point-NeRF can be initialized via direct inference of a pre-trained deep network to produce a neural point cloud; this point cloud can be finetuned to surpass the visual quality of NeRF with 30X faster training time. Point-NeRF can be combined with other 3D reconstruction methods and handles the errors and outliers in such methods via a novel pruning and growing mechanism.

Pipeline


Overview of Point-NeRF. (a) From multi-view images, our model generates depth for each view by using a cost volume-based 3D CNNs and extract 2D features from the input images by a 2D CNN. After aggregating the depth map, we obtain a point-based radiance field in which each point has a spatial location, a confidence and the unprojected image features. (b) To synthesize a novel view, we conduct differentiable ray marching and compute shading only nearby the neural point cloud (e.g., Xa, Xb, Xc). At each shading location, Point-NeRF aggregates features from its K neural point neighbors and compute radiance and volume density then accumulate radiance using density. The entire process is end-to-end trainable and the point-based radiance field can be optimized with the rendering loss..


DTU




NeRF Synthetic



Optimize for 300K steps


   

Optimize for 20K steps



ScanNet


Scene 101


   

Scene 241



Tanks and Temples


Family Scene


   

Truck Scene



Point Growing


Progressively Optimize the inital COLMAP Points


   

Grow Out the Complete Point Cloud from 1000 Points



Paper


Bibtex


@article{xu2022point, title={Point-NeRF: Point-based Neural Radiance Fields}, author={Xu, Qiangeng and Xu, Zexiang and Philip, Julien and Bi, Sai and Shu, Zhixin and Sunkavalli, Kalyan and Neumann, Ulrich}, journal={arXiv preprint arXiv:2201.08845}, year={2022} }