Viewport-Adaptive Scalable Tiling Multicast and Client-Side Neural Enhancement Processing of Live 360-Degree Video

We explore a system for layered multicast streaming of live 360° video to virtual reality (VR) clients characterized with heterogeneous Internet access service classes and 360° video navigation patterns. The system includes scalable 360° tiling processing of the captured content, modeling of the rate-distortion (R-D) characteristics of the compressed 360° tiles, and client-side neural network based enhancement of the delivered viewport. The system is integrated with analysis and optimization of the problem of allocating the network bandwidth of each client class across the streaming 360°content and accounting for the VR clients' navigation patterns such that the expected viewport quality for the streaming session is maximized across the entire client population. We also investigate the benefits and drawbacks of clustering the clients in a class according to their navigation patterns prior to the application of our optimization. We assess the performance of our system and its key components through evaluation experiments and benchmark it against multiple reference methods. Notable performance advances over delivered viewport quality and bandwidth usage efficiency are demonstrated.