VR-Together has undoubtedly gone beyond the state-of-the-art in the field of Social VR, by providing novel and outstanding research and innovation outputs, overcoming existing research challenges and questions, in terms of technology and user experience, respectively. In particular, the project has contributed with two variants of full-fledged Social VR platforms, using off-the-shelf hardware and standard-compliant technology: one lightweight web-based platform; and one native platform that supports extra interactivity features, and different end-users’ representation formats (i.e. Time Varying Meshes, Point Clouds, 3D avatars, 2D windowed ingests from webcam video streams, and ghost users with and without audio communication). These platforms are made up of a set of innovative and modular technological components, like:
· Volumetric capturing systems, based on single and multiple (RGB-D) sensors
· End-to-end low-latency pipelines for the integration of live 2D and volumetric streams, including encoding and distribution solutions.
· Orchestration components for session management.
· Multi-Point Control Unit (MCU) components for an optimized in-cloud processing of Point Clouds and RGB+D streams.
· Web and native media clients supporting the integration of heterogeneous media formats for the end-users’ representation and the virtual scenario, and a set of interactivity features (basic voice control, tele-porting, interaction with objects, events’ handling…).
Beyond the technological components, the project has contributed with professionally content scenarios and assets and new evaluation metrics. This has resulted in open-source and licensed software, open-science datasets, evaluation methodologies and resources, and recommendations on how to provide Social VR experiences. These outputs have been reflected in many (up to 40) publications in high-impact conferences (e.g. ACM CHI, ACM MM, IEEE Virtual Reality) and journals (e.g. IEEE ACCESS, Virtual Reality…) and standardization contributions (e.g. MPEG, ITU, W3C…).
Overall, unlike most of the existing Social VR platforms relying on just the use of synthetic avatars and others requiring more expensive and complex setups for enabling realistic representations (like Microsoft Holoportation), VR-Together has been pioneering on enabling and demonstrating distributed multi-party Social VR experiences with realistic end-users’ representations, including self-representations, in a cost-effective and modular manner. Thus, the project has significantly paved the way towards the adoption of this promising medium, in a set of verticals (entertainment, education, corporate meetings, health…) that will propel distant human communication and interaction to the next levels.
