Attentive Partial Convolution for RGBD Image Inpainting

Inpainting, the technique of reconstructing missing pixels in images, is critical in enhancing image processing and augmented reality (AR) experiences. This paper addresses three prevalent challenges in AR: diminished reality (DR), where unwanted content is removed from the user’s view; the latency in AR head-mounted displays that leads to missing pixels; and the imperfections in depth maps produced by Time-of-Flight (ToF) sensors in AR devices. These challenges compromise the realism and immersion of AR experiences by affecting both texture and geometric integrity of digital content. We introduce a novel Partial Convolution-based framework for RGBD (Red, Green, Blue, Depth) image inpainting that simultaneously restores missing pixels in both the color (RGB) and depth components of an image. Unlike conventional methods that primarily focus on RGB inpainting, our approach integrates depth information, essential for realistic AR applications, by reconstructing the spatial geometry alongside the texture. This dual restoration capability is crucial for creating immersive user experiences in AR by ensuring seamless integration of virtual and real-world elements. Our contributions include the development of an enhanced Partial Convolution model, incorporating attentive normalization and an updated loss function, which significantly outperforms existing models in terms of accuracy and realism in inpainting tasks. This work not only addresses the technical challenges in AR but also opens new avenues for improving image quality in various applications, including online advertising, where the ability to seamlessly edit image content is invaluable.