Periodic Reporting for period 2 - INTUITIVE (INnovative Network for Training in ToUch InteracTIVE Interfaces)
Berichtszeitraum: 2021-10-01 bis 2024-03-31
INTUITIVE was an innovative training network for early stage researchers in the multidisciplinary fields of haptics, haptics technology, neurotechnology, human robotics, robotics, flexible electronics and nanotechnology. The aims were (1) to understand haptics processing within the human brain, paving the ground for designing better user haptics interfaces to be applied in virtual reality and remote operation; (2) to understand how skin biomechanics impact the neural signals generated in a user-interface interaction, again to pave the ground for design; (3) Develop biomimetic signal processing based on tactile primitives and investigate computational functions in typical haptic recognition tasks; (4) Develop biomimetic tactile skin for tomorrow’s touch/haptic and assistive technology; (5) Demonstrate new strategies for robotic haptic manipulation and tactile displays for blind. INTUITIVE also had as a main objective to ensure quick transition of innovation to industry by directly involving key industrial partners, thus leverage the strength of the EU in strategic fields of haptics, flexible electronics and human robotics.
To reach these aims INTUITIVE were composed of academic and industrial partners with a strong competence in relevant subfields. Through a strong recruitment strategy bolstered by the wide networks of individual principal investigators of the partners, INTUITIVE have managed to recruit a strong set of ESRs, inspired to take on the challenges set out by the objectives of the project. In order to foster innovative and independent researchers, INTUITIVE had a strong focus on soft complementary skills such as the process from bringing a lab innovation into a patent, how to pitch a research finding efficiently and other aspects of conducting research and development.
To reach these aims INTUITIVE were composed of academic and industrial partners with a strong competence in relevant subfields. Through a strong recruitment strategy bolstered by the wide networks of individual principal investigators of the partners, INTUITIVE have managed to recruit a strong set of ESRs, inspired to take on the challenges set out by the objectives of the project. In order to foster innovative and independent researchers, INTUITIVE had a strong focus on soft complementary skills such as the process from bringing a lab innovation into a patent, how to pitch a research finding efficiently and other aspects of conducting research and development.
INTUITIVE generated results on multiple fronts related to its objectives. We have achieved a description of haptic input features across a population of cortical neurons, using
multiple, parallel neuron recordings in vivo in the neocortex of the rat. We also developed a new signal processing tool to maximize the yield from such recordings, i.e. increasing the yield in terms of the number of neurons that could be isolated and compared to the activity of other neurons. In this work, we explored the representation of haptic input features between neocortical neurons. We also made significant progress in the study of the principles of how brain state influences the neural representation of haptic input features, achieving controlled manipulation of brain states via visual inputs and recording the impact it had on the interpretation of given tactile input patterns among the cortical neurons. We also achieved a fully functional haptic stimulation transducer for use in rodents, in a physical miniaturized haptic display system that can be used to study haptics processing in rodents as well as other animals including humans. Alongside a macroscopic (‘in-silico’) model of skin tissue dynamics (a high-resolution mass-spring-damper model) we also made significant progress on implementing a corresponding molecular scale model describing how sets of individual mechanoreceptors of the skin are connected to different molecules in the skin and which skin tissue molecules that have which role in creating such skin dynamics, with crucial importance for the amount of information that can be deduced from a haptics interaction. This will in turn provide important insights into materials science approaches to design artificial skin. We implemented a learning classifier system for high-dimensional sensorimotor data that is neuromimetic in that it operates using identification of tactile interaction invariants. INTUITIVE also created what we referred to ‘Next generation sensory augmentation device’, consisting of integration of tactile stimulators in a sensory augmentation device, as well as a usability study for validation of its behavioural benefits as an aid for the blind. We developed a predictive coding model for haptic sensing in rodents to be implemented in haptic sensing for robots. The predictive coding model was implemented for haptic integration of proprioception and tactile information in physical robots engaged in complex haptic exploration (which received a best paper award at IEEE International Symposium on Robotic and Sensors Environments 2024).
Further, INTUITIVE results included the design of an E-skin with microactuator for tactile feedback in robotics and prosthetics, using flexible substrates (e.g. polyimide), as well as fabrication and benchmarking of graphene based touch sensors. We further developed graphene based touch sensors integrated with memristive devices, to achieve a memory effect in the tactile sensing. This relates back to the biological skin model above, in that it uses the memristive devices as a proxy for the rich mechanical dynamics (in effect corresponding to a mechanical state memory) that the biological skin has, and which is used by its sensors to extract the richest possible information from the haptic interaction.
Another interesting domain of results were our softMEMS based stress and slip sensors on ultra-thin chips, for embedding in soft materials, as well as the architectural design in CMOS of a low-power sensory readout and electronic interface. For the integration of sensors on flexible substrates, again to make the sensorized artificial skin more compatible with biological skin, we also worked on solutions for stretchable interconnections and stiff sensor/electronics integration configuration.
INTUITIVE also delivered results on design guidelines for representing audio-tactile graphics for blind users on a two-dimensional display, employing deep learning, and a taxonomy of image processing algorithms in the context of categorizing tactile graphics.
multiple, parallel neuron recordings in vivo in the neocortex of the rat. We also developed a new signal processing tool to maximize the yield from such recordings, i.e. increasing the yield in terms of the number of neurons that could be isolated and compared to the activity of other neurons. In this work, we explored the representation of haptic input features between neocortical neurons. We also made significant progress in the study of the principles of how brain state influences the neural representation of haptic input features, achieving controlled manipulation of brain states via visual inputs and recording the impact it had on the interpretation of given tactile input patterns among the cortical neurons. We also achieved a fully functional haptic stimulation transducer for use in rodents, in a physical miniaturized haptic display system that can be used to study haptics processing in rodents as well as other animals including humans. Alongside a macroscopic (‘in-silico’) model of skin tissue dynamics (a high-resolution mass-spring-damper model) we also made significant progress on implementing a corresponding molecular scale model describing how sets of individual mechanoreceptors of the skin are connected to different molecules in the skin and which skin tissue molecules that have which role in creating such skin dynamics, with crucial importance for the amount of information that can be deduced from a haptics interaction. This will in turn provide important insights into materials science approaches to design artificial skin. We implemented a learning classifier system for high-dimensional sensorimotor data that is neuromimetic in that it operates using identification of tactile interaction invariants. INTUITIVE also created what we referred to ‘Next generation sensory augmentation device’, consisting of integration of tactile stimulators in a sensory augmentation device, as well as a usability study for validation of its behavioural benefits as an aid for the blind. We developed a predictive coding model for haptic sensing in rodents to be implemented in haptic sensing for robots. The predictive coding model was implemented for haptic integration of proprioception and tactile information in physical robots engaged in complex haptic exploration (which received a best paper award at IEEE International Symposium on Robotic and Sensors Environments 2024).
Further, INTUITIVE results included the design of an E-skin with microactuator for tactile feedback in robotics and prosthetics, using flexible substrates (e.g. polyimide), as well as fabrication and benchmarking of graphene based touch sensors. We further developed graphene based touch sensors integrated with memristive devices, to achieve a memory effect in the tactile sensing. This relates back to the biological skin model above, in that it uses the memristive devices as a proxy for the rich mechanical dynamics (in effect corresponding to a mechanical state memory) that the biological skin has, and which is used by its sensors to extract the richest possible information from the haptic interaction.
Another interesting domain of results were our softMEMS based stress and slip sensors on ultra-thin chips, for embedding in soft materials, as well as the architectural design in CMOS of a low-power sensory readout and electronic interface. For the integration of sensors on flexible substrates, again to make the sensorized artificial skin more compatible with biological skin, we also worked on solutions for stretchable interconnections and stiff sensor/electronics integration configuration.
INTUITIVE also delivered results on design guidelines for representing audio-tactile graphics for blind users on a two-dimensional display, employing deep learning, and a taxonomy of image processing algorithms in the context of categorizing tactile graphics.
Despite the pandemic, the research work rapidly accelerated and INTUITIVE became a bustling project with lots of major progress and intense interaction between academia and industry. INTUITIVE centered on developing technology for new types of interfaces, which can for example be applied in the industrial field of metaverse, predicted to be a main driver in the economy many years to come (https://www.cnbc.com/2022/01/01/meta-apple-google-microsoft-gear-up-for-big-augmented-reality-year.html(öffnet in neuem Fenster)). Metaverse is today heavily focused on vision, but the industrial field has also realized how important a component haptics interfaces are for this field. But existing applications are often very coarse/low resolution compared to ongoing work within INTUITIVE. INTUITIVE was also a project featuring deeptech, due to its aims towards an understanding of the brain mechanisms underlying human perception as a key to design new and more intuitive user interfaces, for example for metaverse applications. The haptics technology output can also pave the ground for ground-breaking development in robotics, where haptics can be considered a key for autonomous intelligent system to learn to understand the mechanics of the physical world, an essential component for example for human-robot interactions and for higher general intelligence in robots.