Optimization Based Eye Tracking using Deflectometric Information

Eye tracking is an important tool with a wide range of applications. State-of-the-art eye tracking methods are either reflection-based and track reflections of sparse glint illumination, or appearance-based and exploit 2D features of the acquired eye image. In this contribution, we attempt to significantly improve reflection-based methods by utilizing deflectometric measurements in combination with optimization-based inverse rendering algorithms. Utilizing the known geometry of our deflectometric setup, we develop a differentiable rendering pipeline based on PyTorch3D that simulates a virtual eye under screen illumination. Eventually, we exploit the image-screen-correspondence information from the captured deflectometric measurements to find the eye´s rotation and translation parameters with our renderer via gradient descent. We will demonstrate the working principle of our method in simulation and with real experiments. For the latter, we will show an additional extension of our framework that optimizes the simulated eye shape towards an accurate representation of the real eye, which allows us to accommodate a wide variety of real-world eye shapes in our final method.