We introduce a wave-based image formation model for the problem of non-line-of-sight (NLOS) imaging. Inspired by inverse methods used in seismology, we adapt a frequency-domain method, f-k migration, for solving the inverse NLOS problem.
ABSTRACT
Imaging objects outside a camera’s direct line of sight has important applications in robotic vision, remote sensing, and many other domains. Time-of-flight-based non-line-of-sight (NLOS) imaging systems have recently demonstrated impressive results, but several challenges remain. Image formation and inversion models have been slow or limited by the types of hidden surfaces that can be imaged. Moreover, non-planar sampling surfaces and non-confocal scanning methods have not been supported by efficient NLOS algorithms. With this work, we introduce a wave-based image formation model for the problem of NLOS imaging. Inspired by inverse methods used in seismology, we adapt a frequency-domain method, f-k migration, for solving the inverse NLOS problem. Unlike existing NLOS algorithms, f-k migration is both fast and memory efficient, it is robust to specular and other complex reflectance properties, and we show how it can be used with non-confocally scanned measurements as well as for non-planar sampling surfaces. f-k migration is more robust to measurement noise than alternative methods, generally produces better quality reconstructions, and is easy to implement. We experimentally validate our algorithms with a new NLOS imaging system that records room-sized scenes outdoors under indirect sunlight, and scans persons wearing retroreflective clothing at interactive rates.
RECONSTRUCTING HIDDEN OBJECTS
| Experimental Setup: The imaging system records the time it takes for laser light to scatter off the wall, reflect off the hidden bunny, and return to the wall. By acquiring these timing measurements for different laser positions on the wall, the 3D geometry of the hidden object can be reconstructed. |
| Room-sized Reconstruction: We capture measurements of this hidden room-sized scene and accurately reconstruct the geometry | |
Interactive Capture
| Interactive Capture:By rapidly scanning the wall, we can capture the movement of a person in a retroreflective suit at interactive rates of 4 fps. The reconstruction clearly shows the pose of the actor, imaged from around the corner. |
Outdoor Results
Outdoor Result:We can reconstruct a hidden scene outdoors by scanning the side of a stone building. The stone patterning of the wall is visible in the indirect measurements, but the reconstruction still robustly recovers the geometry.
Hardware Prototype
| Hardware Prototype: Our hardware prototype captures confocal NLOS measurements with a laser that is 10,000 more powerful than previous systems. The sensor generates timestamps of arriving photons at picosecond time intervals, allowing us to precisely localize features of the hidden geometry. |
Press
FILES
- technical paper
- supplemental materials
- presentation slides
- video
Code and Datasets
It took a lot of effort to build and calibrate this hardware setup and to capture these data. Feel free to use the datasets in your own projects, but please acknowledge our work by citing the following papers:
- Matthew O’Toole, Felix Heide, David B. Lindell, Kai Zang, Steven Diamond, and Gordon Wetzstein. 2017. Reconstructing transient images from single-photon sensors. InProc. CVPR. (link)
- Matthew O’Toole, David B. Lindell, and Gordon Wetzstein. 2018. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 7696, 338. (link)
- Felix Heide,Matthew O’Toole, Kai Zang, David B. Lindell, Steven Diamond, and Gordon Wetzstein. 2018. Non-line-of-sight Imaging with partial occluders and surface normals. ACM Trans. Graph. (link)
- David B. Lindell, Gordon Wetzstein, and Matthew O’Toole. 2019. Wave-based non-line-of-sight Imaging using fast f−k migration. ACM Trans. Graph. (SIGGRAPH) 38, 4, 116.(link)
CITATION
David B. Lindell, Gordon Wetzstein, and Matthew O’Toole. 2019. Wave-based non-line-of-sight Imaging using fast f−k migration. ACM Trans. Graph. 38, 4, 116.
BibTeX
@article{Lindell:2019:Wave,
author = {David B. Lindell and Gordon Wetzstein and Matthew O’Toole},
title = {Wave-based non-line-of-sight imaging using fast f-k migration},
journal = {ACM Trans. Graph. (SIGGRAPH)},
volume = {38},
number={4},
pages={116},
year = {2019},
}
Acknowledgements
This project was supported by a Stanford Graduate Fellowship, an NSF CAREER Award (IIS 1553333), a Terman Faculty Fellowship, a Sloan Fellowship, by the KAUST Office of Sponsored Research through the Visual Computing Center CCF grant, the Center for Automotive Research at Stanford (CARS), the DARPA REVEAL program, and a PECASE from the ARO.
Related Projects
You may also be interested in related projects, where we have developed non-line-of-sight imaging systems:
- Metzler et al. 2021. Keyhole Imaging. IEEE Trans. Computational Imaging (link)
- Lindell et al. 2020. Confocal Diffuse Tomography. Nature Communications (link)
- Young et al. 2020. Non-line-of-sight Surface Reconstruction using the Directional Light-cone Transform. CVPR(link)
- Lindell et al. 2019. Wave-based Non-line-of-sight Imaging using Fast f-k Migration. ACM SIGGRAPH (link)
- Heide et al. 2019. Non-line-of-sight Imaging with Partial Occluders and Surface Normals. ACM Transactions on Graphics (presented at SIGGRAPH) (link)
- Lindell et al. 2019. Acoustic Non-line-of-sight Imaging. CVPR (link)
- O’Toole et al. 2018. Confocal Non-line-of-sight Imaging based on the Light-cone Transform. Nature (link)
and direct line-of-sight or transient imaging systems:
- Bergman et al. 2020. Deep Adaptive LiDAR: End-to-end Optimization of Sampling and Depth Completion at Low Sampling Rates. ICCP (link)
- Nishimura et al. 2020. 3D Imaging with an RGB camera and a single SPAD. ECCV (link)
- Heide et al. 2019. Sub-picosecond photon-efficient 3D imaging using single-photon sensors. Scientific Reports (link)
- Lindell et al. 2018. Single-Photon 3D Imaging with Deep Sensor Fusions. ACM SIGGRAPH (link)
- O’Toole et al. 2017. Reconstructing Transient Images from Single-Photon Sensors. CVPR (link)
Reconstruction Procedure
| Measurements and Reconstruction: The captured measurements can be played back as a video, showing light splashing across the wall as it scatters back from the hidden objects (left). Using a wave migration algorithm, the hidden object is recovered from these echoes of light. | |
| Reconstruction Procedure:The reconstruction requires three simple steps. (1) The measurements are Fourier transformed and (2) resampled along the temporal frequency dimension. Then, (3) an inverse Fourier transform recovers the hidden geometry (bottom right). |
Captured Results
| Phasor fields reconstruction | f-k migration reconstruction |
Comparison to Non-line-of-sight imaging using phasor-field virtual wave optics by Liu et al. (2019). The code to run this comparison is available here.
| Bike | |
| Discoball | |
| Dragon | |
| Resolution Chart | |
| Statue | |
Quantitative Results
 |
| Specular Reconstruction:We simulate NLOS measurements and show that the method handles complex reflectance properties better than previous methods such as the Light Cone Transform. |
