Seeing things you don't see: Unifying the philosophy, psychology and neuroscience of multimodal mental imagery

Project Information

STYDS

Grant agreement ID: 726251

DOI

10.3030/726251

Project closed

EC signature date 21 May 2017

Start date 1 September 2017

End date 31 August 2024

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 1 966 530,00

EU contribution

€ 1 966 530,00

Coordinated by

UNIVERSITEIT ANTWERPEN
Belgium

Periodic Reporting for period 5 - STYDS (Seeing things you don't see: Unifying the philosophy, psychology and neuroscience of multimodal mental imagery)

Reporting period: 2023-09-01 to 2024-08-31

Summary of the context and overall objectives of the project

When I am looking at my coffee machine that makes funny noises, this is an instance of multisensory perception – I perceive this event by means of both vision and audition. But very often we only receive sensory stimulation from a multisensory event by means of one sense modality. If I hear the noisy coffee machine in the next room (without seeing it), then how do I represent the visual aspects of this multisensory event?

The aim of this research project is to bring together empirical findings about multimodal perception and empirical findings about (visual, auditory, tactile) mental imagery and argue that on occasions like the one described in the last paragraph, we have multimodal mental imagery: perceptual processing in one sense modality (here: vision) that is triggered by sensory stimulation in another sense modality (here: audition).

Multimodal mental imagery is rife. The vast majority of what we perceive are multisensory events: events that can be perceived in more than one sense modality – like the noisy coffee machine. And most of the time we are only acquainted with these multisensory events via a subset of the sense modalities involved – all the other aspects of these events are represented by means of multisensory mental imagery. This means that multisensory mental imagery is a crucial element of almost all instances of everyday perception, which has wider implications to philosophy of perception and beyond, to epistemological questions about whether we can trust our senses.

Focusing on multimodal mental imagery can help us to understand a number of puzzling perceptual phenomena, like sensory substitution and synaesthesia. Further, manipulating mental imagery has recently become an important clinical procedure in various branches of psychiatry as well as in counteracting implicit bias – using multimodal mental imagery rather than voluntarily and consciously conjured up mental imagery can lead to real progress in these experimental paradigms.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The research on mental imagery in general and on multimodal mental imagery in particular involved a lot of interdisciplinary collaboration, with psychologists and neuroscientists, and the aim of achieiing an understanding of these mental phenomena heavily depended on integrating findings from psychology and neuroscience with philosophy. This general interdisciplinary approach was maintained also in the publications that resulted from the project, including the monograph by the PI on the topic of the ERC project, which was published in 2023 by Oxford University Press. The title of this book is Mental Imagery: Philosophy, Psychology, Neuroscience and it has several book symposia already devoted to it. The interdisciplinarity of the project is also evident in the special issue of Philosophical Transactions of the Royal Society B on the topic of the grant, which was published in 2021 (title: Offline perception), edited by a philosopher (the PI), a psychologist (Peter Fazekas) and a neuroscientidd (Joel Pearson). The results of the research were also communicated to non-expert audiences, for example, by giving talks to large non-specialized audiences, like the Hay-on-Wye festival (UK) or the National Portrait Gallery (UK), by interviews in popular venues like the magazine Der Spiegel (Germany), or GQ (UK), as well as by a TED-Ed video.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

An unexpected outcome of the ERC-funded project was the development of ways in which this research could help with the navigational skills of visually impaired people, by keeping their visual cortices intact (which is achieved by mental imagery training). The PI was awarded an ERC Proof of Concept grant to work on this socially very relevant application of the original research.

Bence Nanay