Periodic Reporting for period 1 - CALIOPE (Co-creativity And Learning: Interactive Opus generation for Piano Education)
Reporting period: 2024-01-01 to 2026-01-31
Summary of the context and overall objectives of the project
The history of music education has seen the use of a variety of technological tools to improve the effectiveness of teaching and, by consequence, the technical ability of musicians. Artificial intelligence is also a tool that could provide such pedagogical benefits, and indeed there has been interest on the use of AI for the betterment of music education at least since 1993, the date of the first workshop on the subject (Smith et al., 1994). Still, music education seems slower than other disciplines in adapting these novel technologies, and there seems to be less research on AI in music education compared to the use of AI in the teaching of other subjects (e.g. computer science, mathematics, language, and medical education). Music education is indeed still largely carried out as it was before the personal computer became a common item in the average household, the most common setting being one-to-one sessions with a teacher, often one hour per week, complemented by solo practice the student does to prepare for the following lesson. The study material for both the lessons and the solo practice at the beginner level is typically taken from “method” books, i.e. a collection of exercises and short musical pieces of progressive difficulty that a composer/pedagogist curated as a way to gradually introduce new techniques and challenges to the student. One downside of these books is that they are a one-size-fits-all solution, lacking the ability to adapt to different learner’s characteristics, strengths, and shortcomings.
This project tries to change this status by embedding AI systems into the educational practice. In particular, the benefit we envision as achievable by using AI system within education is the betterment of the personalisation of the educational experience. As stated above, most of music education in Western-classical environments is done by using method books and fixed curricula. A teacher could search for additional exercises to give to their students and could even write some new ones in some cases, but this can be extremely expensive in terms of time for the teacher, and doing so for each student could quickly prove overwhelming. Notably, the fact that some teachers do try to go these lengths is a testament to how valuable the generation of new exercises could be seen as a valuable teaching tool.
Personalisation within an educational context is not merely a flourish added on top of a curriculum or a quirk of some more extroverted teachers. Education research has shown that personalising the learning approach and material can have a tremendous impact on the quality of education. The difference is so stark that this effect is known as the “two-sigma effect” (Bloom, 1984), because it has been shown that a personalised education can lead to results that are two standard deviations away from a control group not using personalised approaches. While we don’t expect to be able to obtain such results by the use of AI exercise generation alone, we strongly believe this tool would make it much more realistic for teachers to structure their educational sessions in a personalised manner, which in turn should allow them to achieve this two-sigma improvement.
Smith, M., Smaill, A., Wiggins, G. A., and Van Rijsbergen, C. J., editors (1994). Music Education: An Artificial Intelligence Approach: Proceedings of a Workshop Held as Part of AI-ED 93, World Conference on Artificial Intelligence in Education, Edinburgh, Scotland, 25 August 1993. Workshops in Computing. Springer, London.
Bloom, B. S. (1984). The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Educational Researcher, 13(6):4–16.
This project tries to change this status by embedding AI systems into the educational practice. In particular, the benefit we envision as achievable by using AI system within education is the betterment of the personalisation of the educational experience. As stated above, most of music education in Western-classical environments is done by using method books and fixed curricula. A teacher could search for additional exercises to give to their students and could even write some new ones in some cases, but this can be extremely expensive in terms of time for the teacher, and doing so for each student could quickly prove overwhelming. Notably, the fact that some teachers do try to go these lengths is a testament to how valuable the generation of new exercises could be seen as a valuable teaching tool.
Personalisation within an educational context is not merely a flourish added on top of a curriculum or a quirk of some more extroverted teachers. Education research has shown that personalising the learning approach and material can have a tremendous impact on the quality of education. The difference is so stark that this effect is known as the “two-sigma effect” (Bloom, 1984), because it has been shown that a personalised education can lead to results that are two standard deviations away from a control group not using personalised approaches. While we don’t expect to be able to obtain such results by the use of AI exercise generation alone, we strongly believe this tool would make it much more realistic for teachers to structure their educational sessions in a personalised manner, which in turn should allow them to achieve this two-sigma improvement.
Smith, M., Smaill, A., Wiggins, G. A., and Van Rijsbergen, C. J., editors (1994). Music Education: An Artificial Intelligence Approach: Proceedings of a Workshop Held as Part of AI-ED 93, World Conference on Artificial Intelligence in Education, Edinburgh, Scotland, 25 August 1993. Workshops in Computing. Springer, London.
Bloom, B. S. (1984). The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Educational Researcher, 13(6):4–16.
Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far
At the inception of the project, to assess exactly how commonly AI is used for music education and its personalisation, we performed a systematic literature review . The systematic review retrieved 360 article of which 40 were retained for analysis after exclusion and inclusion criteria application. The results showed that indeed, while there seems to be interest in both the use of AI for music education and the betterment of its personalization through computational means, it seems that the explicit generation of exercises is not currently being implemented. Some approaches that do exist limit themselves to retrieving material from existing databases, or generating complementary aspects like visualizations or feedback, rather than the educational material itself.
The project continued with more practical research addressing this topic. One important task in any music system for music education is the evaluation of difficulty. Indeed difficulty estimation is now a somewhat common task in Music Information Retrieval, but difficulty is often reduced to a single number. we experimented with the definition of multiple difficulty aspects that we can compute from a digital version of a music score, including Fingering displacement (defined as the distance one’s fingers must move from their estimated resting positions to play each note), Playing speed (defined as the inverse of the inter-onset interval (IOI) between each note), and Polyphony (defined as the ratio of chord counts (or beats) to non-chord counts (or beats), computed separately for various chord sizes). The only available datasets use single scalar values for difficulty, but as a validation for these features a Gradient Boost model (XGBoost) was trained to predict a 9-class difficulty rating based on the newly designed features. The predictions with this model achieved a MAE of 0.7920 suggests that even if there are wrong estimates the estimated label on the 9-class scale is not too far off from the ground truth. The system also leveraged the classifier to generate reductions of music with a specific level of difficulty.
A different approach to the personalised modelling of difficulty is to cognitively model the expected difficulty of a piece, using a model that would allow to adapt this expectation to the different users of the system. In particular, there has been some evidence in prior literature that Information Content correlates to difficulty. we believe that Information Content is a valid measure to be used to model the cognitive load of the various elements on a score, which make up the difficulty of a piece: seen before elements will be easier than those that have never been seen, and more common note sequences will be easier than rare note sequences. For further putting this idea to the test, we leveraged IDyOMs, a modelling system using markovian processed to build statistical expectation, to construct distributions of pitch, duration and fingering annotations. The system was trained on a dataset using three difficulty classes (Beginner, Intermediate, Professional). The trained model yields probability estimates for all the elements in the analysed pieces.
A statistical test (Kruskal-Wallis H test, followed by a Dunn post-hoc test) confirmed that the information about pitch, duration and fingering results in information content distributions that are significantly different between difficulty levels. This confirms the hypothesis that information content is correlated to the concept of difficulty, confirming prior research, but our model is also capable of using the learnt statistical models to generate novel music of the desired difficulty level.
The studies went on to adapt the same approach to the specific modelling of different students, as a way to test how well this cognitive model can offer an account of personalised difficulty. We named this personalised difficulty "Challenge", defined as "how difficult a certain piece will be to a certain person at a certain stage in their development". The above mentioned IDyOM was once again used to assess if we can model how the level of challenge decreases when a learner has progressed through a defined curriculum of studies. We trained the system on different subsets of Mikrokosmos, a method book by Bela Bartok, and studied the information content of the harder pieces from that collection. As expected, a model trained on more exercises has a lower information content for difficult pieces. An additional experiment tried to model different learning paths, using the same Mikrokosmos, as one possible learning path and a small dataset of serialist pieces as a different path. As expected, when facing a difficult piece in a serialist style, the model trained on the small serialist dataset outperformed the model trained on the bigger but less specific Mikrokosmos dataset, showing that the cognitive approach to the definition of Challenge that we used can indeed model personalised learning paths.
The project continued with more practical research addressing this topic. One important task in any music system for music education is the evaluation of difficulty. Indeed difficulty estimation is now a somewhat common task in Music Information Retrieval, but difficulty is often reduced to a single number. we experimented with the definition of multiple difficulty aspects that we can compute from a digital version of a music score, including Fingering displacement (defined as the distance one’s fingers must move from their estimated resting positions to play each note), Playing speed (defined as the inverse of the inter-onset interval (IOI) between each note), and Polyphony (defined as the ratio of chord counts (or beats) to non-chord counts (or beats), computed separately for various chord sizes). The only available datasets use single scalar values for difficulty, but as a validation for these features a Gradient Boost model (XGBoost) was trained to predict a 9-class difficulty rating based on the newly designed features. The predictions with this model achieved a MAE of 0.7920 suggests that even if there are wrong estimates the estimated label on the 9-class scale is not too far off from the ground truth. The system also leveraged the classifier to generate reductions of music with a specific level of difficulty.
A different approach to the personalised modelling of difficulty is to cognitively model the expected difficulty of a piece, using a model that would allow to adapt this expectation to the different users of the system. In particular, there has been some evidence in prior literature that Information Content correlates to difficulty. we believe that Information Content is a valid measure to be used to model the cognitive load of the various elements on a score, which make up the difficulty of a piece: seen before elements will be easier than those that have never been seen, and more common note sequences will be easier than rare note sequences. For further putting this idea to the test, we leveraged IDyOMs, a modelling system using markovian processed to build statistical expectation, to construct distributions of pitch, duration and fingering annotations. The system was trained on a dataset using three difficulty classes (Beginner, Intermediate, Professional). The trained model yields probability estimates for all the elements in the analysed pieces.
A statistical test (Kruskal-Wallis H test, followed by a Dunn post-hoc test) confirmed that the information about pitch, duration and fingering results in information content distributions that are significantly different between difficulty levels. This confirms the hypothesis that information content is correlated to the concept of difficulty, confirming prior research, but our model is also capable of using the learnt statistical models to generate novel music of the desired difficulty level.
The studies went on to adapt the same approach to the specific modelling of different students, as a way to test how well this cognitive model can offer an account of personalised difficulty. We named this personalised difficulty "Challenge", defined as "how difficult a certain piece will be to a certain person at a certain stage in their development". The above mentioned IDyOM was once again used to assess if we can model how the level of challenge decreases when a learner has progressed through a defined curriculum of studies. We trained the system on different subsets of Mikrokosmos, a method book by Bela Bartok, and studied the information content of the harder pieces from that collection. As expected, a model trained on more exercises has a lower information content for difficult pieces. An additional experiment tried to model different learning paths, using the same Mikrokosmos, as one possible learning path and a small dataset of serialist pieces as a different path. As expected, when facing a difficult piece in a serialist style, the model trained on the small serialist dataset outperformed the model trained on the bigger but less specific Mikrokosmos dataset, showing that the cognitive approach to the definition of Challenge that we used can indeed model personalised learning paths.
Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)
The project has outlined novel approaches to the personalisation of music education, introducing the concept of Challenge and providing a cognitive approach to its estimation, as well as exploring systems for the estimation of difficulty that have incremental betterments over the state of the art.
These results need to be more thoroughly tested with actual students and teachers, which in turn requires the engineering of more readily-usable systems that embed the results of the research, and to fully integrate exercise generation with the challenge framework introduced by the project.
Summarizing, the project has:
- investigates systematically the state of the art, underlining research gaps;
- improved upon existing difficulty estimation adding exercise generation on top of them;
- designed and tested a new approach based on the concept of Challenge for the personalisation of music education.
Following-up on these points will potentially lead to commercial systems which could change our approach to music education.
These results need to be more thoroughly tested with actual students and teachers, which in turn requires the engineering of more readily-usable systems that embed the results of the research, and to fully integrate exercise generation with the challenge framework introduced by the project.
Summarizing, the project has:
- investigates systematically the state of the art, underlining research gaps;
- improved upon existing difficulty estimation adding exercise generation on top of them;
- designed and tested a new approach based on the concept of Challenge for the personalisation of music education.
Following-up on these points will potentially lead to commercial systems which could change our approach to music education.