Periodic Reporting for period 3 - MISEL (MULTISPECTRAL INTELLIGENT VISION SYSTEM WITH EMBEDDED LOW-POWER NEURAL COMPUTING)
Período documentado: 2024-01-01 hasta 2025-06-30
The MISEL project aimed at a low-power bio-inspired vision system with data processing based on cellular, cerebellar and cortical processors providing adaptivity, learning and reasoning. With in-sensor spatio-temporal neuromorphic computing based on complex events enabling a fast and energy-efficient operation, the system targeted advanced situation awareness with small size, weight and power consumption (SWaP), outperforming conventional state-of-the-art (SoA) systems.
To achieve this objective, the project has developed all crucial components of the system:
- adaptive multi-band (VIS-to-NIR) pixels for the camera. The pixels combine silicon diodes with colloidal quantum dot/metal-insulator-graphene (CQD-MIG) diodes monolithically fabricated on top of the CMOS sensing-and-computing layer to work on multiple wavelengths.
- in-sensor computing for data reduction and adaptation. Local computing drives sensor adaptation for signal enhancement, while event-based operation reduces output data stream.
- dense FeRAM monolithically integrated on top of CMOS computing layer, used for synaptic communication and plasticity in CNN block.
The project demonstrated effectiveness of the approach, which stems from organizing computing hierarchically similarly to biology:
- cellular sensor-processor (CSP) for near-sensor processing – sensors and processors form cells and neighboring cells interact like in the retina.
- cerebellar processor for fast spatio-temporal processing integrated with the CSP on the same chip providing motion pattern analysis as well as coarse object recognition capabilities using content addressable memories and analysis of event patterns in space and time.
- cortical processor based on CNN and HDC for high-level processing, learning, and prediction capabilities with feedback to cellular/cerebellar processor.
In the MISEL system, hardware and algorithms were co-designed to find the best possible trade-off between complexity and performance.
To achieve this objective, the project has developed all crucial components of the system:
- adaptive multi-band (VIS-to-NIR) pixels for the camera. The pixels combine silicon diodes with colloidal quantum dot/metal-insulator-graphene (CQD-MIG) diodes monolithically fabricated on top of the CMOS sensing-and-computing layer to work on multiple wavelengths.
- in-sensor computing for data reduction and adaptation. Local computing drives sensor adaptation for signal enhancement, while event-based operation reduces output data stream.
- dense FeRAM monolithically integrated on top of CMOS computing layer, used for synaptic communication and plasticity in CNN block.
The project demonstrated effectiveness of the approach, which stems from organizing computing hierarchically similarly to biology:
- cellular sensor-processor (CSP) for near-sensor processing – sensors and processors form cells and neighboring cells interact like in the retina.
- cerebellar processor for fast spatio-temporal processing integrated with the CSP on the same chip providing motion pattern analysis as well as coarse object recognition capabilities using content addressable memories and analysis of event patterns in space and time.
- cortical processor based on CNN and HDC for high-level processing, learning, and prediction capabilities with feedback to cellular/cerebellar processor.
In the MISEL system, hardware and algorithms were co-designed to find the best possible trade-off between complexity and performance.
In WP1, CQD-MIG diode arrays and ferroelectric memory arrays were integrated on top of CMOS circuits in back end of line (BEOL) process. First, vias were etched and filled to gain contacts to the CMOS circuits, and then metal layer structures were added to serve as bottom electrodes for the post-processed devices. For CQD-MIG sensor the thickness of that layer was optimized for graphene deposition. Sets of PbS, PbS/CdS, and lead-free InAs nanoparticles were synthesized and their application on the MIG diodes was improved by structuring the nanoparticle layer. Images taken by the CQD-MIG sensor array show photodetection capabilities. For FeRAM devices, the bottom electrode required development of a special process to achieve high-stress TiN film with specific crystal orientation to enhance the ferroelectricity of the next layer. The integration of ferroelectric Hf0.5Zr0.5O2 (HZO) on CMOS was performed in BEOL and characterized with focus on memory state dynamics. The different area ratios between the HZO capacitor and the CMOS yield different dynamics, enabling application-driven solutions. Analog memory states were achieved with pulsed writing methods, showing good retention (measured to 1e3 sec., extrapolated to >1 year). Full arrays of CMOS FeFETs were implemented and are currently under measurements. According to simulations, combining CMOS and FeFETs can provide CNN weights with 5-bits resolution. The FeFET integration process was transferred from ULUND to VTT, where a proof-of-concept FeFETs were integrated. VTT has now the capability to integrate FeCaps on 8” wafers. Good performance and high wafer uniformity has been demonstrated on 6” wafers and similar device tests are already planned for 8” wafers.
In WP2 neuromorphic circuits that gain performance from massive parallelism and computing topology were implemented. Several integrated circuits were taped-out (jointly with WP3) including CQD-MIG sensor readout circuit, associative memory, convolution accelerator, and HDC arrays. Cellular and cerebellar processors were put on the same chip to take advantage of high-speed on-chip data buses. The sensor front-end accommodates data path for both CMOS and CQD-MIG pixels. The processing hardware provides temporal difference events and fixed-pattern noise (FPN)-compensated intensity images. This data is used to generate complex events that compactly describe spatial and temporal features. The complex events are used for interest point detection, optical flow computation, and motion pattern extraction using an associative processor. On-chip cellular neural net provides means for segmentation, clustering, and region of interest (ROI). A chip-to-chip link between cellular/cerebellar and cortical processors collects and transfers ROI data. On-chip controllers (custom and RISC-V) support large-scale real-time system operation and communication.
WP3 tackled higher-level information processing and system integration. The designed cortical processor, comprising a DNN for region of interest (ROI) detection and feature extraction, and a HDC model for sequence classification, shows a very low power consumption. The chip controlled by on-chip RISC-V contains two data paths: one consisting of a DNN and an HDC blocks in CMOS technology, and the other comprises a DNN combining CMOS with FeFETs and a CMOS-only HDC block. Studies on gesture recognition have shown that hypervectors with 2048 components and DNNs with 5 bits of resolution provide sufficient accuracy. This eases hardware requirements, and thus its implementation as an edge device. A digital twin of the system supports hardware-realistic algorithm evaluation and co-design in WP2, WP3, and WP4.
WP4 evaluated the sensor layer, the computational layer, and the entire system in selected applications. The use-case was defined in parallel to the development of the components, taking into account predicted hardware capabilities and legal issues. Evaluation criteria were defined, an evaluation corpus (dataset) was developed, and a comparative evaluation was done between algorithms applied on the original corpus data and the data from HW-realistic sensor emulator.
In WP2 neuromorphic circuits that gain performance from massive parallelism and computing topology were implemented. Several integrated circuits were taped-out (jointly with WP3) including CQD-MIG sensor readout circuit, associative memory, convolution accelerator, and HDC arrays. Cellular and cerebellar processors were put on the same chip to take advantage of high-speed on-chip data buses. The sensor front-end accommodates data path for both CMOS and CQD-MIG pixels. The processing hardware provides temporal difference events and fixed-pattern noise (FPN)-compensated intensity images. This data is used to generate complex events that compactly describe spatial and temporal features. The complex events are used for interest point detection, optical flow computation, and motion pattern extraction using an associative processor. On-chip cellular neural net provides means for segmentation, clustering, and region of interest (ROI). A chip-to-chip link between cellular/cerebellar and cortical processors collects and transfers ROI data. On-chip controllers (custom and RISC-V) support large-scale real-time system operation and communication.
WP3 tackled higher-level information processing and system integration. The designed cortical processor, comprising a DNN for region of interest (ROI) detection and feature extraction, and a HDC model for sequence classification, shows a very low power consumption. The chip controlled by on-chip RISC-V contains two data paths: one consisting of a DNN and an HDC blocks in CMOS technology, and the other comprises a DNN combining CMOS with FeFETs and a CMOS-only HDC block. Studies on gesture recognition have shown that hypervectors with 2048 components and DNNs with 5 bits of resolution provide sufficient accuracy. This eases hardware requirements, and thus its implementation as an edge device. A digital twin of the system supports hardware-realistic algorithm evaluation and co-design in WP2, WP3, and WP4.
WP4 evaluated the sensor layer, the computational layer, and the entire system in selected applications. The use-case was defined in parallel to the development of the components, taking into account predicted hardware capabilities and legal issues. Evaluation criteria were defined, an evaluation corpus (dataset) was developed, and a comparative evaluation was done between algorithms applied on the original corpus data and the data from HW-realistic sensor emulator.
Logarithmic sensor front ends offer many desirable features but suffer from very large FPN. New online calculation means were developed to remove the FPN from the intensity image. It enables generating of spatial and spatio-temporal events for more powerful and flexible analysis than with conventional temporal difference events. This allows for additional event modalities, offering higher information content and better reduction of data streams, and thus new edge-computing applications benefiting from complex events.
As part of the cellular/cerebellar processor, the largest cellular binary processor in the world with asynchronous propagation network and local memories was implemented and evaluated.
To meet the constraints of low-resource edge-computing systems, a novel sequence-classification method capable of handling real-world tasks was developed. It integrates neural-based representation extraction with a hyperdimensional (HD) data analysis module. To further improve data analysis, a novel concept of HD sequence classification inspired by the Hidden Markov Model paradigm was formulated and evaluated.
Future generations of neuromorphic vision systems with low SWaP, like MISEL, would be useful on-board of small autonomous vehicles (drones, robots) working in complex harsh environments.
As part of the cellular/cerebellar processor, the largest cellular binary processor in the world with asynchronous propagation network and local memories was implemented and evaluated.
To meet the constraints of low-resource edge-computing systems, a novel sequence-classification method capable of handling real-world tasks was developed. It integrates neural-based representation extraction with a hyperdimensional (HD) data analysis module. To further improve data analysis, a novel concept of HD sequence classification inspired by the Hidden Markov Model paradigm was formulated and evaluated.
Future generations of neuromorphic vision systems with low SWaP, like MISEL, would be useful on-board of small autonomous vehicles (drones, robots) working in complex harsh environments.