CONnected through roBOTS: physically coupling humans to boost handwriting and music learning

Project Information

CONBOTS

Grant agreement ID: 871803

DOI

10.3030/871803

Project closed

EC signature date 5 December 2019

Start date 1 January 2020

End date 31 December 2023

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 4 810 796,25

EU contribution

€ 4 810 796,25

Coordinated by

UNIVERSITY OF NEWCASTLE UPON TYNE
United Kingdom

Periodic Reporting for period 3 - CONBOTS (CONnected through roBOTS: physically coupling humans to boost handwriting and music learning)

Reporting period: 2022-09-01 to 2023-12-31

Summary of the context and overall objectives of the project

The project tackles the challenge of introducing a new class of robots, i.e. the CONBOTS, in education scenarios. The CONBOTS will physically couple people to facilitate learning and to improve efficiency during training of new motor skills. The CONBOTS platform will be designed and tested to be used in two different contexts:
a) Children learning to handwrite.
b) Beginner musicians learning to play violin or drums.

The CONBOTS project will impact on society in different ways: i) spreading the adoption of robots to a novel application scenario, i.e. education; ii) fostering the growing area of Education 4.0; iii) having children as target end-users and thus fostering children capability to work at the intersection of people and technology.

CONBOTS project is build aroung the following project objectives:
A) To define the user needs and platform requirements and to deploy measures to raise awareness and take-up of CONBOTS scientific and technological achievements;
B) To design physically interacting robotic haptic devices in the form of an end-effector robotic workstation and a wearable exoskeleton;
C) To exploit recent neuroscience insights on physical interaction and modern control theory to establish a Game Theory of physical communication for improving human-human and human-robot interactions;
D) To design a bi-directional user interface with Augmented Reality serious games, wearable sensors and instrumented objects for maximising the impact of physical interaction in learning contexts;
E) To develop machine learning algorithms for describing the physical, emotional, and mental state of the users, in order to tailor serious games and optimise the training process to the specific user needs;
F) To Integrate the different CONBOTS enabling technologies and optimise them for the two application scenarios;
G) To test the efficacy of CONBOTS platforms for training sensorimotor skills of children during handwriting and music learning.

The CONBOTS project successfully managed to pursue its scientific and technical challenges by demonstrating the validity of the CONBOTS platform for its use in relevant application scenarios, i.e. handwriting and music playing.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Hereinafter, relevant activities carried out in each WP during the first 32 months towards the achievement of the above objectives are briefly summarized:
WP1: kick-off the project, organization of regular meetings among partners, submissions and approvals of Ethics committee documentations for preliminary experiments, mitigation of the effect of COVID-19 pandemic on project activities, request of amendment to the European Commission, reshaping of the project’s timeline, addition of Newcastle University as new coordinator unit.
WP2: definition of the user needs and requirements of the CONBOTS platform, technical and functional specifications of the different platform components, as well as for their integration; analysis of the non-technical specifications arising from the ethical, legal, socio-economic, privacy and cybersecurity issues related to the use of the CONBOTS technologies.
WP3: design of haptic robotic devices for human-human physical communication, i.e. end-effector haptic device for handwriting learning and exoskeleton devices for music learning
WP4: study of the physiological mechanisms of human-human physical communication, i.e. how performance and perception resulting from interaction with a robotic partner compare with the interaction between humans, how human-human interaction changes in a 3DoF task considering gravity and asymmetric interaction, and what are the effects of delay on haptic communication.
WP5: realization of the bi-directional user interface, which encompasses AR/VR based serious games, wearable sensors, and instrumented objects. Software components of the AR environment were developed to facilitate children learning in the two application scenarios, while initial experiments with adult violin players have been performed to test how suitable is the VR for this application scenario.
WP6: release of the machine learning models to estimate the physical and mental state of the user.
WP7: integration of the different technologies of the CONBOTS platform and its customization for the two application scenarios. Definition of the general architecture of the platform, and first implementation of the developed technologies, that include wearable sensors, robotic haptic devices, a graphical user interface, AR-based software environment developed in Unity3D, and instrumented objects.
WP8: planning and executing preliminary experiments in the two application scenarios, i.e. handwriting and music learning. As regards handwriting, twelve children were recruited and evaluated using the GHEE (Grapho-motor Handwriting Exercise & Empowerment) protocol. For the music scenario, another exploratory study has been setup and carried out to test the use of AR for violin education. Moreover, 10 drum students were followed up for 1 year, every 2 months, in the laboratory environment of GU. In every lesson, a fixed set of exercises and a play-along drum sequence was taught by the teacher and performed by the students, while audio, video and MoCap data of the students were collected.
WP9: dissemination activities and the activities related to the exploitation of project results.
WP10: submission of the deliverables related to the Ethics issues highlighted in the Ethics Summary Report.

CONBOTS research activities have achieved all of the main goals of the project, from technology to experiments, including ethical and legal aspects. In particular, we have addressed privacy and cybersecurity issues, validated the machine learning model in the use cases, and validated the integrated CONBOTS platform in the relevant scenarios of violin playing and handwriting through a number of experimental studies. CONBOTS empirical results demonstrated the effectiveness of robot mediated haptic feedback in relevant musical tasks (i.e. playing violin) and handwriting. CONBOTS research output can be summarized in the following numbers: 31 deliverables submitted, 10 milestones achieved, 65+ publications in journals/conference proceedings, 50+ invited talks/workshops/other public events, 20+ new experimental protocols designed with data from 300+ subjects gathered, 5 new hardware prototypes, 3 AR/VR apps, 5 software tools, 9 PhD fellowships, and 9 post-docs promoted to lecturer/assist prof positions.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

CONBOTS’ major progresses beyond the state of the art are:

1) Design and test of a haptic device conceived of as a system capable to exchange haptic forces among users.
2) Design and test of a portable upper-limb exoskeleton for learning to play violin.
3) Customization of a non-invasive, easy-to-use, instrumented wearable system, capable to measure physiological signals of arousal of users.


CONBOTS results are expected to impact in different ways, such as:

1) Strengthening European excellence in Robotics S&T
2) Boosting the use of robotics in promising application areas
3) Lowering barriers in the deployment of robotics-based solutions

CONBOTS technologies used in the two application scenarios: handwriting and violin playing.