How Many Views Are Needed to Reconstruct an Unknown Object Using NeRF?




Authors:

S. Pan, L. Jin, H. Hu, M. Popović, M. Bennewitz

Type:

Conference Proceeding

Published in:

IEEE International Conference on Robotics and Automation (ICRA)

Year:

2024

Related Projects:

AID4Crops - Automation and AI for Monitoring and Decision Making of Horticultural Crops

DOI:

https://doi.org/10.1109/ICRA57147.2024.10610617

Links:

PreprintPublicationVideoCode

BibTex String

@inproceedings{pan2024icra,
title={How many views are needed to reconstruct an unknown object using nerf?},
author={Pan, Sicong and Jin, Liren and Hu, Hao and Popovi{\'c}, Marija and Bennewitz, Maren},
booktitle={IEEE International Conference on Robotics and Automation (ICRA)},
year={2024}
}
Topic

Abstract:

Neural Radiance Fields (NeRFs) are gaining significant interest for online active object reconstruction due to their exceptional memory efficiency and requirement for only posed RGB inputs. Previous NeRF-based view planning methods exhibit computational inefficiency since they rely on an iterative paradigm, consisting of (1) retraining the NeRF when new images arrive; and (2) planning a path to the next best view only. To address these limitations, we propose a non-iterative pipeline based on the Prediction of the Required number of Views (PRV). The key idea behind our approach is that the required number of views to reconstruct an object depends on its complexity. Therefore, we design a deep neural network, named PRVNet, to predict the required number of views, allowing us to tailor the data acquisition based on the object complexity and plan a globally shortest path. To train our PRVNet, we generate supervision labels using the ShapeNet dataset. Simulated experiments show that our PRV-based view planning method outperforms baselines, achieving good reconstruction quality while significantly reducing movement cost and planning time. We further justify the generalization ability of our approach in a real-world experiment.