Since autopilot was first demonstrated in 1914, pilots have found themselves increasingly “connected” to their aircrafts through numerous instruments in the cockpit. These tools have been introduced to assist pilots with the aim of increasing safety. Ironically, today there is a risk that pilots find it difficult to cope with the vast amounts of information generated by the instruments. Therefore, to continue to pioneer safety, improvement and optimization of situational awareness is needed. Maturing current technologies as well as new developments are required to meet the needs of next generation aircraft. These will result in increased crew efficiency and enhanced flight safety. Employing a smart cockpit interface, fully adapted to the pilots’ need, is a way to meet these objectives.
Led by CSEM, the PEGGASUS consortium has created a system for human–machine interface for cockpit avionics, towards the development of new generation cockpits. Using the latest in artificial intelligence (AI) and computer vision technologies, this project created remote eye-gaze tracking and gesture recognition for pilots in a single framework. Its purpose is to enhance human-machine interaction for today’s complex flight operations to improve safety, crew efficiency and pilot training. Such systems need to remain accurate and robust during a flight, considering aviation-specific environmental factors such as extreme lighting conditions and vibrations. Also, the use of eye tracking in combination with measures like skin conductance was investigated in the context of crew monitoring.
To design a powerful and tailored solution, the consortium partners provided the essential complementary skills, ensuring the project’s success. CSEM developed the vision system and machine learning algorithms for gaze tracking and gesture recognition, addressing the unique challenges and requirements in the aeronautic context. SERMA Ingénierie was responsible for integrating the PEGGASUS system HW into a cockpit demonstrator for testing. ETH Zurich, with their expertise in the field of human-machine interaction and aviation safety, performed the design and execution of a simulator study with ten pilots from SWISS International Airlines. Additionally, SWISS International Airlines supported the consortium partners with the expertise of a professional airline on the use cases and insights for future training possibilities.