Periodic Reporting for period 3 - EXPERIENCE (The “Extended-Personal Reality”: augmented recording and transmission of virtual senses through artificial-IntelligENCE)
Berichtszeitraum: 2024-01-01 bis 2025-06-30
What do you think would happen if on social media, in addition to photos, videos and posts, we could share real experiences and emotions? Imagine a ride on a roller coaster or a visit to a museum or, perhaps, a friend’s trip to Tokyo that becomes our trip to Tokyo.
EXPERIENCE (The “Extended-Personal Reality” : augmenting recording and transmission of virtual senses through artificial intelligence) is a European project included among the Future and Emerging Technologies of Horizon 2020 regarding the use of artificial intelligence in the social sciences and neurosciences. The project is coordinated by the University of Pisa, and sees the participation of the University of Siena, the University of Padua, the University of Rome “Tor Vergata”, Polytechnic University of Valencia, Karolinska Institutet, French CEA in Paris, and CSEM Swiss Center for Microelectronics.
The EXPERIENCE consortium aims to widespread the use of virtual reality in real life. Indeed, while the generation of virtual environments is only for highly specialised individuals, through EXPERIENCE the public at large should be able to easily create virtual environments as they create photos and videos. To this extent, the consortium aimed to build a scientific and technological framework to compensate the significant differences in evoked psychological, cognitive, neurophysiological, and behavioural responses that may exist between actual physical environments and their virtual simulations.
Throughout the project, a novel wearable monitoring system was developed that can record the visual scene synchronised with neurophysiological signals. In addition, an easy-to-use software tool was created to automatically generate a virtual reality environment with scene understanding (i.e. recognition of scanned objects). The technology also allows for modification of the virtual environment to add or alter objects, potentially to induce different arousing, pleasant or unpleasant emotional states. Biofeedback and manipulation of the user's space-time perception were also achieved through the numerous scientific studies conducted during the project.
EXPERIENCE (The “Extended-Personal Reality” : augmenting recording and transmission of virtual senses through artificial intelligence) is a European project included among the Future and Emerging Technologies of Horizon 2020 regarding the use of artificial intelligence in the social sciences and neurosciences. The project is coordinated by the University of Pisa, and sees the participation of the University of Siena, the University of Padua, the University of Rome “Tor Vergata”, Polytechnic University of Valencia, Karolinska Institutet, French CEA in Paris, and CSEM Swiss Center for Microelectronics.
The EXPERIENCE consortium aims to widespread the use of virtual reality in real life. Indeed, while the generation of virtual environments is only for highly specialised individuals, through EXPERIENCE the public at large should be able to easily create virtual environments as they create photos and videos. To this extent, the consortium aimed to build a scientific and technological framework to compensate the significant differences in evoked psychological, cognitive, neurophysiological, and behavioural responses that may exist between actual physical environments and their virtual simulations.
Throughout the project, a novel wearable monitoring system was developed that can record the visual scene synchronised with neurophysiological signals. In addition, an easy-to-use software tool was created to automatically generate a virtual reality environment with scene understanding (i.e. recognition of scanned objects). The technology also allows for modification of the virtual environment to add or alter objects, potentially to induce different arousing, pleasant or unpleasant emotional states. Biofeedback and manipulation of the user's space-time perception were also achieved through the numerous scientific studies conducted during the project.
The EXPERIENCE project achieved significant scientific, technological, and dissemination milestones. The EXPERIENCE Wearable System was devised in the first 18 months of the action, allowing for the automatic creation of virtual reality environments through tablet and/or wearable camera, supported by a performing laptop in a backpack. In addition, ad-hoc electronics was devised to synchronise camera and physiological data recordings from brain and heart to estimate the subject’s emotional state while scanning the scene. The final system was validated technologically and scientifically, integrating hardware, strong data encryption, and compliance with safety standards. Immersive VR environments compatible with fMRI were deployed and tested in real neuroimaging contexts, while a modular AI-driven 3D reconstruction pipeline was completed, enabling scene editing and real-time physiological biofeedback.
Extensive neuroscientific studies were conducted, including hundreds of healthy individuals as well as participants experiencing depression, revealing new neurophysiological markers of time perception via neural and cardiovascular signals. Research on cybersickness also demonstrated novel approaches to its reduction through neuromodulation, highlighting potential clinical applications. A unified multimodal analysis framework was developed, combining neuroimaging and wearable data, and state-of-the-art results were achieved in neural-activity-based semantic image reconstruction using advanced AI models.
Clinical translation was supported by the development of a diagnostic VR system that classified depressive symptoms with up to 80% accuracy, alongside the validation of two VR-based therapeutic protocols (HRV biofeedback and time perception recalibration), both surpassing clinical effectiveness thresholds.
From an innovation and impact perspective, six Key Exploitable Results were identified, with an exploitation roadmap and market opportunity analysis securing potential industrial engagement. Dissemination was strong and impactful, with more than a hundred peer-reviewed publications in scientific journal articles and papers published in conference proceedings, the achievement of six major milestones, and active visibility through conferences, workshops, and public science events.
Extensive neuroscientific studies were conducted, including hundreds of healthy individuals as well as participants experiencing depression, revealing new neurophysiological markers of time perception via neural and cardiovascular signals. Research on cybersickness also demonstrated novel approaches to its reduction through neuromodulation, highlighting potential clinical applications. A unified multimodal analysis framework was developed, combining neuroimaging and wearable data, and state-of-the-art results were achieved in neural-activity-based semantic image reconstruction using advanced AI models.
Clinical translation was supported by the development of a diagnostic VR system that classified depressive symptoms with up to 80% accuracy, alongside the validation of two VR-based therapeutic protocols (HRV biofeedback and time perception recalibration), both surpassing clinical effectiveness thresholds.
From an innovation and impact perspective, six Key Exploitable Results were identified, with an exploitation roadmap and market opportunity analysis securing potential industrial engagement. Dissemination was strong and impactful, with more than a hundred peer-reviewed publications in scientific journal articles and papers published in conference proceedings, the achievement of six major milestones, and active visibility through conferences, workshops, and public science events.
EXPERIENCE may be exploited in many real-life contexts; for example, to evaluate the socio-psychometric characteristics (mood, character tendencies, attitudes, etc.) of a subject, highlighting any psychiatric pathologies. If the manipulation of personal virtual reality allows you to stimulate specific neuro-cardiovascular reactions, inducing particular cognitive-behavioural and emotional states in a subject, it goes without saying that the system can also be used to treat very common pathologies, such as depression, anxiety and stress.
The consortium has also performed a thorough market analysis to widen the EXPERIENCE exploitation avenues in areas other than health, such as gaming, e-learning, and digital tourism.
The consortium has also performed a thorough market analysis to widen the EXPERIENCE exploitation avenues in areas other than health, such as gaming, e-learning, and digital tourism.