Multi-sensory, User-centred, Shared cultural Experiences through Interactive Technologies

Project Information

MuseIT

Grant agreement ID: 101061441

DOI

10.3030/101061441

Project closed

EC signature date 2 June 2022

Start date 1 October 2022

End date 31 December 2025

Funded under

• Culture, creativity and inclusive society

Total cost

€ 3 358 745,00

EU contribution

€ 3 358 745,00

Coordinated by

HOEGSKOLAN I BORAS
Sweden

Periodic Reporting for period 1 - MuseIT (Multi-sensory, User-centred, Shared cultural Experiences through Interactive Technologies)

Reporting period: 2022-10-01 to 2023-09-30

Summary of the context and overall objectives of the project

With around 16% of the world population living with some form of disability (as estimated by WHO) and the many barriers to accessibility and participation that are embedded in the societal structures, the MuseIT project underscores the pressing need for a more inclusive society as a direct response to the European and global calls for human rights and equal opportunities for all.

The MuseIT project aims to address the challenges of inclusion and broader access to cultural assets through various scientific and innovation activities that will help to preserve and safeguard cultural heritage in an inclusive way. Participatory co-design activities will be integral to enable users with disabilities to be involved in decisions and designs that will affect their lives. Enriched experiences of cultural assets created by MuseIT will not be limited to this group and is likely to enhance the level of participation and enjoyment for all people. The main objective is to promote Inclusion, Participation, Accessibility and Equal Opportunities for All through the development of a platform for remote co-creative engagement with cultural assets and experiences.

The project aims to improve:
• Accessibility of cultural assets by development of multisensory representations and alternative expressions to enable engagement by the public regardless of functional or sensory impairments.
• Engagement with cultural assets and cultural co-creation by a broader public.
• Methodologies for preservation and safeguarding of cultural heritage with inclusion at its core.

Impacts are likely to include the democratization of cultural asset experiences, and tangible growth in creative and cultural industries.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Within the first period of the MuseIT project:

WP1:
Different boards in the project were established.
Different administrative tools and services were put in place and routines and codes of conducts were established.
Based on requests and decisions made in the MuseIT PMB, an amendment process was initiated and changes to the consortium compositions were made.

WP2:
a) Premises for end-user engagement in participatory co-design activities were established, including development of study protocols, various informed consent forms, information material in various formats (e.g. in different languages, easy read format, etc), creating various fora for engagement, establishing trust, and forming long-term relationships with the users, and more.
b) A survey of previous research on the needs for digital cultural assets of people with disabilities and experiences of cultural institutions was conducted.
c) Studies of user needs and experiences (with regard to MuseIT technical requirements) were carried out in two ideation workshops.

WP3:
a) A survey on datasets and resources for multimodal representations, including: multimodal knowledge graph datasets, KG completion, KG embeddings, and related techniques.
b) A new technique for learning knowledge graph embeddings exclusively based on paths, with comparable performance to the state of the art and 3 orders of magnitude less parameters
c) A framework for explainable embeddings for finding quality dimensions
d) Activities around the use of large language models (LLMs) for knowledge and ontology engineering, including a best paper award winner at ISWC and the organisation of an international Hackathon attended by 40+ PhD students from 10+ institutions

WP4:
a) Definition of user needs and system requirements regarding the virtual experiences in cultural heritage. Completion of the Task 4.1 through the submission of the deliverable D4.1.
b) Recruitment of digital cultural heritage assets (2D and 3D) that will be used in the virtual exhibition of T4.3 and the completion of the museology design of the virtual exhibition.
c) Definition of draft integration plan of the MuseIT web platform with the overall MuseIT architecture. Preliminary investigation of accessibility requirements for the web interface. First draft of login page and landing page.

WP5:
a) The design of the low latency architecture has been completed. This was delivered to the project in month 8 as D5.1 Co-creation Services for Shared (and Intangible) Cultural Engagements.
b) Procedures have been streamlined, including a browser-based interface connecting autonomously to the low latency architecture server (JackTrip). A video channel has also been added to the Jack Trip platform.
c) Informal experiments have taken place in preparation for a combined audio and sensor channel. We have been pursuing different options for a haptic channel. Informal experiments have taken place in relation to biosensors, biodata and audification.
d) Participatory workshops have provided useful user information for the design of an accessible, interactive dashboard for multisensory musical composition for all (T5.4)

WP6:
Created low level ontology, which serves as the foundational structure for the project's cultural assets and has the potential for extension to incorporate certain existing ontologies and formal specifications. Also constructed a Demo that envelops a general pipeline including both components of the knowledge graph as well as the interaction with user's emotions and music generation

WP7:
a) Full mapping of MuseIT software components
b) Preliminary definition of transfer protocols
c) First definition of MuseIT architecture plan
d) Integration with CERTH services

WP8:
a) Creation and continuous update of the MuseIT IP Registry,
b) Identification of the Key Exploitable Results,
c) Defined an initial exploitation plan for MuseIT technical outcomes (D8.1)

WP9:
Three ethics experts were identified and appointed as the external advisers in the MuseIT Ethics Advisory Board.
EAB meetings were held and various ethical aspects of the project were discussed.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

At this early stages of the project different methodologies and techniques are still being explored and experimented with, hence more items will be added here as the research in the project matures. For now, just a couple of items that have been tested are:

• The design and functionality of the tool for creation of meaning bearing haptic patterns has been extended and tested with potential user, proving advances in terms of function, use areas and user-friendliness.
• Eearly results show that our new technique for KG embeddings on paths performs comparably or better than the state of the art at the task of knowledge graph relation prediction, but costing three orders of magnitude less compute power and memory.

Project Logo