IMage BAsed Landing Solutions

Project Information

IMBALS

Grant agreement ID: 785459

DOI

10.3030/785459

Project closed

EC signature date 12 February 2018

Start date 1 March 2018

End date 31 December 2022

Funded under

• SOCIETAL CHALLENGES - Smart, Green And Integrated Transport

Total cost

€ 2 509 922,14

EU contribution

€ 1 919 852,49

Coordinated by

SCIOTEQ
Belgium

Periodic Reporting for period 3 - IMBALS (IMage BAsed Landing Solutions)

Reporting period: 2021-03-01 to 2022-12-31

Summary of the context and overall objectives of the project

The approach and landing phases are the most critical ones in commercial aircraft operations and many aircraft accidents have a significant contribution from human error. Hence, automation during these phases is a promising path to increase aviation safety, which is a societal interest. The overall objective of the IMBALS project is to develop, validate and verify a certifiable Image Processing Platform (IPP) and demonstrate it in a Vision Landing System (VLS) that is capable of autolanding a large passenger aircraft based on images supplied by an on-board camera system.
The workload reduction provided by fully automated landing is one of the enablers to transition to single-pilot operations, which would be a workaround for an expected pilot shortage after recovery from the Covid-19 pandemic. This will help to address the societal challenge towards mobility.
The project objective was to reach TRL5.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The following work has been performed in the project:

WP1 Detailed project plan (closed)
A master plan has been approved in a formal planning review and serves the project management in WP9.

WP2 Definition of requirements
The functional requirements have been defined by Airbus starting from the CONOPS for automated landing. From these, equipment level requirements were derived. These requirements have been iterated on with feedback from the V&V planning, system definition and validation.

WP3 Validation & verification plan
The V&V activities were planned in the V&V plan. This plan identified features for the IPP prototypes that were needed in support of the V&V activities. The plan also identified the needed V&V environment, including two dedicated test benches.

WP4 System definition
KU Leuven demonstrated a series of representative algorithm that detect the runway in the images and extract a pose estimation out of that. Based on this algorithm, (UN)MANNED defined a set of vision operators inside their Sol Language to support the porting of the algorithm to Sol platform and the integration into their target hardware. (UN)MANNED defined the end-to-end algorithm using those vision operators. This was the basis for validation work in WP5.
Meanwhile KU Leuven further developed its algorithm toolbox with more capabilities (incl. processing images from a visual wavelength as well as from an infra-red wavelength camera and a tracking algorithm). (UN)MANNED extended its Sol vision library with KU Leuven’s new operators in order to iterate on the current application.
ScioTeq compared different system and equipment architectures. ScioTeq conducted the preliminary system safety assessment to select the best fit architecture. ScioTeq defined various interfaces and communication protocols in close collaboration with Airbus and the partners.
ScioTeq defined the technical specification and concept for the test bench software.

WP5 System validation
UNM demonstrated their end-to-end algorithm in Sol (defined in WP4) on their embedded target hardware (SolAero). This enabled a first measurement of the latency, identification of bottlenecks and resolution of the major bottlenecks. The on-target processing latency is a key characteristic closely tracked throughout the project.
KU Leuven has iteratively evaluated the performance of their algorithms.
ScioTeq prototyped the main building blocks for the validation prototypes and started their validation.
ScioTeq integrated all the technology bricks into IPP prototypes and validated their interoperability. Then ScioTeq, KU Leuven and (UN)MANNED went through an interative and collaborative process of testing, refining the algorithm, porting the algorithm and retesting. This resulted in a series of releases, of which several were delivered to Airbus for testing on their DISCO Bench.
Every iteration in the validation made progress in the performance of the IPP and at the same time revealed new issues and challenges. In november 2022, the iterations had to be stopped because the project was running to its end. This was at a point where the IPP operated as specified from 6 km till the flare. Solutions for supporting touch down and roll out had already been identified but could not be implemented anymore.

WP6 System verification
ScioTeq conducted per requirement verification on the configuration resulting from WP5. This verification made the capabilities and the limitations of the technology and the test benches clear. These were analyzed for resolution and recommendations are formulated towards future R&D for advancing the state-of-art.

WP7 Final report
The final report captures a summary of the technical results and a plan to exploit the results up to commercialization of certified IPP products by ScioTeq and UNM.

WP8 Prestudies
WP8 can be divided in two main activities: collection of video footage and a technology survey.
TEKEVER recorded visual and Infra-Red video footage during flight patterns that were defined by TEKEVER and ScioTeq. Airbus provided complementary visual wavelength and fused VIS/IR video footage. This footage is a stimulus during the testing of algorithms.
The technology survey yields a "shopping list" with building blocks and principles that could be used in the upcoming work packages and this is captured in D8.2 "Technology Survey Report".

WP9 Project management & dissemination
ScioTeq coordinated the project within the consortium, with Clean Sky 2 JU and with Airbus.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The main progress beyond the state of art relates to the increased autonomy that will be reached for the landing of large passenger aircraft. About 99% of landings with commercial aircraft still include a phase where the human pilot takes full control of the aircraft. With 49% of fatal aircraft accidents happening in the approach and landing phases and many of them having a demonstratable contribution from human error, it is clear that fully automated landing will have a positive impact on aviation safety, provided that commensurate safety levels of the automated system can be assured. To reach this goal, the image based landing function must meet the safety objectives for catastrophic failure conditions. This immediately sets the highest safety levels to the IPP and its hosted algorithms. This will require novel techniques to establish a measurable confidence level about the output of these algorithms. There is no prior art in certifying image processing algorithms with possibly catastrophic failure conditions for use in commercial aircraft.
The high level of automation enabled with image based landing will also support the transition from dual to single pilot operation in large commercial aircraft, yielding an important competitive advantage for the aircraft manufacturer and airliner.
The project has reached TRL4. It was close to reaching TRL5, but TRL5 could not be claimed. For this there has not been sufficient testing with real video footage with reliable meta-data and the usage domain of the current IPP is to limited.

Examples of perception errors

Demo of runway detection and pose estimation

Example of position error versus ground truth

Concept of testbench for simulation and replay of prerecorded footage