Optimized deep learning algorithms for application with data from PMD cameras

Photonic mixing device (PMD) cameras are well suited for fast and robust three-dimensional image acquisition. Large pixel sizes, however, severely limit their spatial resolution. Thus, their low-resolution (LR) 3D output images (depth and intensity images) can benefit significantly from super-resolution (SR) algorithms. In the past few years numerous learning based algorithms have been suggested for super-resolution in the context of artificial intelligence. In this context, depth image SR typically uses complementary high-resolution (HR) intensity or color images to enhance spatial resolution. Besides data fusion approaches, several deep learning based single image SR techniques are available. Both variants show promising results that outperform classic SR methods. We present a new deep learning SR approach for related intensity and depth images captured via time-of-flight (ToF) imaging without the need of an additional HR image. First, a PMD camera’s LR intensity image is enhanced by a single image SR algorithm based on a convolutional neural network. Its HR intensity output is then used to increase depth image’s spatial resolution with a guided deep multi-scale SR algorithm.