Point-NeRF: Point-based Neural Radiance Fields

(CVPR 2022 Oral 🤩🤯🔥)

A high performance novel neural radiance representation.

Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu
Kalyan Sunkavalli , Ulrich Neumann

Paper Code

Oral Presentation @ CVPR 2022


Volumetric neural rendering methods like NeRF generate high-quality view synthesis results but are optimized per-scene leading to prohibitive reconstruction time. On the other hand, deep multi-view stereo methods can quickly reconstruct scene geometry via direct network inference. Point-NeRF combines the advantages of these two approaches by using neural 3D point clouds, with associated neural features, to model a radiance field. Point-NeRF can be rendered efficiently by aggregating neural point features near scene surfaces, in a ray marching-based rendering pipeline. Moreover, Point-NeRF can be initialized via direct inference of a pre-trained deep network to produce a neural point cloud; this point cloud can be finetuned to surpass the visual quality of NeRF with 30X faster training time. Point-NeRF can be combined with other 3D reconstruction methods and handles the errors and outliers in such methods via a novel pruning and growing mechanism.


Overview of Point-NeRF. (a) From multi-view images, our model generates depth for each view by using a cost volume-based 3D CNNs and extract 2D features from the input images by a 2D CNN. After aggregating the depth map, we obtain a point-based radiance field in which each point has a spatial location, a confidence and the unprojected image features. (b) To synthesize a novel view, we conduct differentiable ray marching and compute shading only nearby the neural point cloud (e.g., Xa, Xb, Xc). At each shading location, Point-NeRF aggregates features from its K neural point neighbors and compute radiance and volume density then accumulate radiance using density. The entire process is end-to-end trainable and the point-based radiance field can be optimized with the rendering loss..


NeRF Synthetic

Optimize for 300K steps


Optimize for 20K steps


Scene 101


Scene 241

Tanks and Temples

Family Scene


Truck Scene

Point Growing

Progressively Optimize the inital COLMAP Points


Grow Out the Complete Point Cloud from 1000 Points



@article{xu2022point, title={Point-NeRF: Point-based Neural Radiance Fields}, author={Xu, Qiangeng and Xu, Zexiang and Philip, Julien and Bi, Sai and Shu, Zhixin and Sunkavalli, Kalyan and Neumann, Ulrich}, journal={arXiv preprint arXiv:2201.08845}, year={2022} }