Haptic perception in interaction with other senses

Hannah B. Helbig and Marc O. Ernst


Human perception is inherently multisensory: we perceive the world simultaneously with multiple senses. While strolling the farmers market, for example, we might become aware of the presence of a delicious fruit by its characteristic smell. We might use our senses of vision and touch to identify the fruit by its typical size and shape and touch it to select only that one with the distinctive soft texture that signals ‘ripe’. When we take a bite of the fruit, we taste its characteristic flavour and hear a slight smacking sound which confirms that the fruit we perceive with our senses of vision, touch, audition, smell and taste is a ripe, delicious peach. That is, in the natural environment the information delivered by our sense of touch is combined with information gathered by each of the other senses to create a robust percept. Combining information from multiple systems is essential because no information-processing system, neither technical nor biological, is powerful enough to provide a precise and accurate sensory estimate under all conditions. In this chapter, we address the question how the human brain combines sensory information in order to form a robust, reliable and coherent percept of the world around us. Most of the theoretical considerations we discuss here are not confined to interactions of touch with the other senses but generalise to interactions among the other senses (e.g., visual-auditory). Therefore, we keep these considerations general but focus on examples demonstrating how haptics is combined with other sensory modalities.
Information that is perceived through different sensory pathways can be qualitatively different: the senses can provide either complementary or redundant information. Redundant sensory signals provide information about the same sort of object property (e.g., size) and are represented in a common frame of reference, in the same units. To stay with the example of the peach, vision and touch provide redundant information about the peach’s size or shape. In contrast, vision and taste provide complementary information about the identity of the object.
In general, we benefit from integrating multiple sources of information. Combining complementary information is advantageous because it extends the range and variety of what can be perceived from any one sense in isolation [1] and can reduce perceptual ambiguity. Furthermore, integrating multiple sensory sources usually leads to improved perceptual performance, more precise judgments and enhances detection of stimuli (e.g., [2–11]).