Collaborative Research: CNS Core: Medium: miVirtualSeat: Semantics-aware Content Distribution for Immersive Meeting Environments

The COVID-19 pandemic has greatly reinforced the need for virtual meetings in both work-related and social settings. It has also highlighted the dire lack of video conferencing tools that can simulate the rich immersive experience of in-person meetings since the current tools often leading to fatigue caused by having to interact over a non-ideal communication medium. This collaborative project brings together investigators from University of Illinois, Urbana-Champaign, University of Massachusetts at Amherst, and New Jersey Institute of Technology to research, build, and evaluate a distributed system, called miVirtualSeat, that more closely simulates the immersive experience of in-person meetings, including physical and virtual participants in a physical meeting space.

The project is focused on key research challenges in providing an immersive meeting experience where physical and virtual participants interact with each other in a physical meeting room. Some participants are virtually present in the physical meeting room, but physically located in remote sites with only limited compute and network resources. The challenges are (a) detecting, tracking, and localizing distributed physical and virtual 360-degree avatars and objects in the joint immersive scene in real-time, (b) reducing the bandwidth and latency of delivering integrated and synchronized 360-degree, volumetric, and 2D/3D video, and ambisonics audio, and (c) ensuring good quality-of-experience in the form of natural interaction between physical and virtual participants.

The project addresses an immediate and important need for a post-pandemic society to enable immersive hybrid meetings that arise in the context of classrooms, conferences, office meetings, and social gatherings. miVirtualSeat will enable these meetings with a physical meeting room and remote sites situated at each of the investigators' institutions. The outcome of the project will be new undergraduate and graduate courses in the emerging field of advanced mixed reality immersive environments. Through outreach activities, the project members will showcase miVirtualSeat and expose the broader public to the capabilities of distributed AR/VR systems.

Information and results from this project will be maintained on the project website that is available to the public at https://monet.cs.illinois.edu/miVirtualSeat/. These will include publications, links to software repositories, media releases, and links to system data repositories derived from the project.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.