Experimental Interaction I: Multisensory Design

Classically the senses have been described as our windows to the world and limited to the Aristotelian 5 senses (sight, touch, taste, smell and hearing). In the modern era, we have not only expanded this list of modalities, but radically redefined what it means to perceive, incorporating a deeper understanding of embodied experience, multimodal integration, and the interplay between natural and technologically mediated sensory systems. Consequentialism and utilitarianism have argued that the ability to feel, by affording the possibility of both suffering and pleasure, is the heart of all ethics and morality. Phenomenology has argued that perception is not a passive reception of sensory data but an active, embodied engagement with the world, where meaning arises through our lived experiences and the interplay between our body, consciousness, and environment. Posthumanism has argued that the boundaries of perception, cognition, and identity are not fixed but fluid, challenging the anthropocentric view of the senses and embracing a vision of human experience as deeply interconnected with technology, non-human entities, and broader ecological systems. The Predictive Coding framework has argued that what we feel and sense is our world: an ongoing controlled hallucination, with the brain constantly trying to reduce prediction error through multimodal sensory data and its expected probability.

In this module, we will investigate the ways in which technology could/will/does change the nature of the senses, for humans and non-human agents. Can the artificial modification and enhancement of our senses give rise to new experiences, identities, and selves? Can these advancements foster inclusivity and accommodate the diverse abilities and sensory perceptions of individuals? And what does it mean for machines to have the ability to feel and is this even possible?

The module begins with experiments with the strange psychophysical phenomena of crossmodal and multimodal experiences, informed by the Predictive Coding framework. Students are invited to reflect and practically investigate how sensory information gives rise to secondary phenomenological experiences, such as desires and emotions, and how these, in turn, might drive both consumerism and maladaptive behaviours. These explorations will lead to “Provotypes”, or Provocative Prototypes that envision and demonstrate new and peculiar forms of sense in and through technological objects.

The Provotypes will address one of three subtopics:

Connecting / Immersing the Senses in Digital Worlds
Expanding / Augmenting Human Senses
Giving Senses / Embodiment to Non-Humans and AI

Text by Luke Franzke