Automated Measurement of Engagement Level of Children with Autism Spectrum Conditions during Human-robot Interaction

Project Information

EngageME

Grant agreement ID: 701236

DOI

10.3030/701236

Project closed

EC signature date 11 March 2016

Start date 1 October 2016

End date 30 September 2019

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 239 860,80

EU contribution

€ 239 860,80

Coordinated by

UNIVERSITAET AUGSBURG
Germany

Periodic Reporting for period 2 - EngageME (Automated Measurement of Engagement Level of Children with Autism Spectrum Conditions during Human-robot Interaction)

Reporting period: 2018-10-01 to 2019-09-30

Summary of the context and overall objectives of the project

Engaging children with ASC (Autism Spectrum Conditions) in communication centred activities during educational therapy is
one of the cardinal challenges by ASC and contributes to its poor outcome. To this end, therapists recently started using
humanoid robots (e.g. NAO) as assistive tools. However, this technology lacks the ability to autonomously engage with
children, which is the key for improving the therapy and, thus, learning opportunities. Existing approaches typically use
machine learning algorithms to estimate the engagement of children with ASC from their head-pose or eye-gaze inferred
from face-videos. These approaches are rather limited for modeling atypical behavioral displays of engagement of children
with ASC, which can vary considerably across the children. The main objective of EngageME is to bring novel machine learning models that can for the first time effectively leverage
multi-modal behavioural cues, including facial expressions, head pose, vocal and physiological cues, to realize fully automated
context-sensitive estimation of engagement levels of children with ASC. EngageME brings novel
technology/models for endowing assistive robots with ability to accurately ‘sense’ engagement levels of children with ASC
during robot-assisted therapy, while providing the candidate with a set of skills needed to become one of the frontiers in the
emerging field of affect-sensitive assistive technology.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The researcher has implemented the activities outlined in Part B of the proposal, covering the reporting period. All the milestones have successfully been met.
The summary of the contributions:
- technical:
• the researcher designed and implemented a novel framework named personalized machine learning, which is currently the state of the art approach for multimodal modelling of behavioural expressions of children with autism recorded during therapy sessions with a robot and a therapist (WP2-3);
• the researcher pre-processed the multi-modal (audio-visual-physiological) data of the child-robot interactions, being part of the multi-cultural dataset used for automated engagement estimation of the children with autism (WP1);
• the researcher participated in collection of new child-robot interactions in schools for a period of 3 months (8 weeks). These data were not originally planned in the proposal, but make additional contribution to the description of work in the proposal.

- dissemination:
• the researcher organized three workshops at top tier computing and robotics conferences:
The 3rd Workshop on Affective Computing (AC) at Int'l Joint Conference on Artificial Intelligence (IJCAI), 2019
Workshop on Personalized Machine Learning for Future Health, Members Event, MIT Media Lab 2018
The 1st Int'l Workshop on Deep Affective Learning and Context Modeling (DAL-COM), CVPR 2017
The Special Session on Affective Robots, ROMAN 2017
The 1st Int'l Workshop on Affective Computing for Social Robotics (ACSR), in conj. with ROMAN 2016

• the researcher gave 5 invited talks at the top tier computing/robotics conferences and universities
• the researcher created the project website
• the researcher published the research carried out in 15 research articles (conference and journal)
• the researcher performed a demo of the newly designed robot perception technology at science fairs in Serbia, Japan and USA.

- teaching:
• the researcher created a graduate course for teaching Personalized Machine Learning, which, together with Prof. Picard, he taught as the main lecturer at MIT Media Lab during the Spring semester in 2017 (January-May).

- supervision:
• the researcher supervised 4 Master thesis projects at MIT Media Lab and mentored over 10 undergraduate students who participated in the work done as part of EngageMe.
• during the return stage, the researcher has supervised 5 master students’ theses at Augsburg University/Imperial College; two of those have achieved the “distinguished thesis” award (top 5% of all master theses at the department).

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The researcher has successfully finished the deliverables promised in the project and carried out the target dissemination activities promised in the proposal. During the return stage at Augsburg University, the researcher has worked on translating the knowledge and expertise from MIT, and over the period of 2 years while there. The project has significantly increased the awareness of the feasibility of the automated robot technology for autism therapy, and through media and news this was well communicated to the public. The project has been selected as the success story by the European Commission. This fellowship has helped the researcher to obtain a research position in one of the top tech companies where he will continue his research on multimodal personalized AI for healthcare, thus broadening the developed methods beyond the autism application.

- The work carried out in EngageME has been accepted as the first research article on Human-robot interaction in the most prestigious journal in the field, Science Robotics. This raised a lot of awareness about the importance of the technology that is being developed as part of this project. This work also raised societal awareness (through news media) of challenges with autism, and the need for better assistive technologies for autism. This contributes towards European policy objectives and strategies and policy making.
- EngageME has been selected as the Success Story by European Commission.
- The results of EngageME have been published in over 20 research articles at top conferences (uploaded on the EC System).
- EngageME was disseminated world-wide by news media
- This all enhanced the researcher’s visibility in the research community, which opened lots of potential opportunities for the next step.
- As the next step, the researcher is considering to become a lecturer in one of top European universities as the next part of his career
- The carried work has large potential to enhance future markets for social robots by the means of developed machine learning algorithms and tools.
- The integrated EngageME models (together with NAO robot) were evaluated/disseminated in several London/Boston Schools with kids with autism and typically developing kids. The overall feedback from the kids and teachers was very positive and they were open to using such technology in their daily activities.
- The dissemination mentioned above showed that the designed models are also applicable to typically developing children, which makes them useful for perception of every-day-robots that can be deployed in regular schools as well.

EngageME logo