At this stage, the project has progressed on three main activities: the iterative studies for the collection of needs and specifications, the hardware development, and the software development (incl. the AI algorithms).
For the iterative design of the ABILITY solution, we have finalised a number of user studies to gather initial input for the design process. We have collected valuable information and insights to both inspire and direct our design work (and described in deliverables from WP2). In total, 104 individual representatives of the future users, of whom 97 had a visual impairment (including 6 persons who were deafblind), participated in interviews, demonstrator workshops or lo-fi workshops to gather information about current practices for accessing digital information, challenges, needs and wishes for specifications, at four different locations (Sweden, France, Lithuania and Germany) from experts to end-users with various backgrounds and age groups.
On the hardware side, we tested various solutions for the integration of the localised multitouch feedback approach both with OLED screens and the engraved cover glasses of the insideONE tablet. We demonstrated the ability to vibrate those screens and have successfully integrated the localised multitouch technology with an OLED display for a functional prototype.
Regarding the actuators themselves, to facilitate the integration and increase the reliability in the future product, we developed an innovative patch technology able to provide piezoelectric patches that can be integrated collectively onto the OLED screen.
For the braille display, we tested a first design of the pins, and obtained satisfactory results in terms of required force and displacement and holding force. This was validated with a prototype actuating two braille cells. New pin designs are currently investigated to increase the holding force to satisfy users’ requirements while keeping in mind the future industrial process for assembly. Studies also revealed the need for touch detection on the display, for cursor routing or interaction gestures, thus solutions are also being investigated to satisfy these requirements.
In parallel, the software of the tablet has been upgraded for the integration of the future capabilities. A new version of the InsideONE tablet has been released early 2024 with boosted capabilities, such as using the latest Intel Core i7 processor, with up to 1TB storage with a high-speed SSD, improved battery life, using a USB-C power supply, a stereo audio system with four speakers, 2 cameras with buil-in OCR, and all operating on Windows 11.
On the AI algorithms side, initial investigations on the AI algorithms best suited for image and content analysis, and predictive typing were conducted both through a literature review and testing of algorithms on sample images and texts. This led to the development of an image analysis pipeline to demonstrate and test different AI-based methods for image analysis with automatic image captioning, free question and answering on images, images segmentation, object detection, face detection, saliency and edge detection; as well as methods and tools to extract text from images and converters to transform text to braille and braille to text as well as text to speech. Initial work has also been conducted on implementing the AI-based next word prediction to assist typing with multimodal interaction. Finally, through the interviews and workshops with experts and target users, potential use cases were also preliminary explored.
