Listening through the body

What makes an experience immersive? Drawing on my practice across large-scale installations, moving image, light, sound, data, and AI, this keynote explores how visual, spatial, auditory, and bodily elements can be composed to make invisible systems emotionally and physically perceptible. Immersion, I suggest, is more than sensory spectacle: it can create spaces for attention, reflection, and renewed human agency.

Understanding how humans perceive their biological self and the physical world is a fundamental question in neuroscience. Recent technologies raise a further question: how flexibly can the brain accept and represent an altered self? To address this issue, we developed an independently controllable artificial “sixth finger.” Unlike conventional supernumerary robotic limbs that are controlled by movements of existing body parts, our system uses co-contraction EMG signals that do not produce overt movement, enabling voluntary control of the artificial finger independently of other body movements. Our previous experiments showed that approximately one hour of adaptation can induce behavioral and perceptual changes consistent with a sense of ownership toward the artificial finger. Furthermore, fMRI measurements before and after adaptation suggest changes in brain activity. These findings suggest that an additional artificial body part can be embodied and alter neural representations of the body. I will also discuss possibilities of neural augmentation and the information-encoding capacity of the human brain.

In this talk, we explores the frontier of embodied interaction by presenting a suite of novel, targeted haptic interfaces designed to stimulate various parts of the human body, fundamentally enhancing user presence and immersion. We will begin with an explorative project supported by Hong Kong Environmental Protection Department for a full-body immersive Typhoon experience in VR. We then pin-point to individual body parts. Starting with the face, we introduce VirCHEW Reality, a face-worn haptic device that uses pneumatic actuation to provide kinesthetic force feedback, simulating the mechanical properties of food for virtual dining experiences. Moving to the hands, we will discuss ViboPneumo, a finger-worn vibratory-pneumatic device capable of dynamically increasing or decreasing the perceived texture roughness of physical surfaces in mixed reality. We will also dive into advanced skin-level innovations with our Skin-Integrated Multimodal Haptic Interface. Alongside it, we will present ThermOuch, a wearable thermo-haptic device that safely leverages the thermal grill illusion to induce non-harmful pain sensations, elevating arousal and realism in VR. For the lower extremities, we will introduce PropelWalker, a calf-worn, propeller-based system equipped with ducted fans that simulates buoyancy and fluid resistance, allowing users to physically feel the weight of walking through water, mud, or sand. Collectively, these projects demonstrate how geographically targeted, multimodal haptic technologies—spanning from facial kinetics to leg-based force feedback—can profoundly elevate body ownership, sensory realism, and engagement in virtual worlds.